Cross-modal model training method, image cross-modal conversion method and electronic equipment

Abstract

The invention provides a cross-modal model training method. The method comprises the following steps: acquiring a first modal image and a second modal image corresponding to the first modal image; training an encoder in a preset auto-encoder by using the first modal image to obtain an encoder model of the encoder; according to the second modal image and a training feature matrix output by the encoder to the first modal image, training a preset adversarial network to obtain an adversarial model corresponding to the adversarial network, the adversarial model comprising a generator model; and combining the encoder model and the generator model to obtain a cross-modal model. By using the method, the generator model obtained by training can effectively perform cross-modal conversion on images with different view angles and expression modes. The invention further provides an image cross-modal conversion method and electronic equipment.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a cross-modal model training method, an image cross-modal conversion method, and an electronic device. Background Art

[0002] In the field of medical technology, the image information obtained by a single-modality imaging device often cannot fully reflect the complex characteristics of the disease. In order to accurately judge the condition, it is necessary to obtain medical images of multiple modalities to assist doctors in diagnosing the condition, for example, using magnetic resonance images and positron emission tomography images to assist in the diagnosis of Alzheimer's disease. In related technologies, the acquisition of medical images of different modalities requires a lot of time and financial resources, and it is impossible to convert the content in the medical images, making it difficult to achieve cross-modal conversion of images. Summary of the invention

[0003] In view of the above, it is necessary to provide a cross-modal model training method, an image cross-modal conversion method and an electronic device that can solve the technical problem of difficulty in achieving cross-modal conversion of images.

[0004] The present application provides a cross-modal model training method, the method comprising: obtaining a first modal image and a second modal image corresponding to the first modal image, the first modal image and the second modal image having different imaging perspectives, using the first modal image to train an encoder in a preset autoencoder to obtain an encoder model of the encoder, training a preset adversarial network based on the second modal image and a training feature matrix output by the encoder for the first modal image to obtain an adversarial model corresponding to the adversarial network, the adversarial model comprising a generator model, and combining the encoder model and the generator model to obtain a cross-modal model.

[0005] In some embodiments of the present application, the autoencoder includes the encoder and the decoder, and the use of the first modality image to train the encoder in the preset autoencoder to obtain the encoder model of the encoder includes: calling the encoder to perform feature extraction on the first modality image to obtain the training feature matrix, and calling the decoder to reconstruct the training feature matrix to obtain a reconstructed image, calculating the encoding loss value of the autoencoder based on the first modality image and the reconstructed image, and adjusting the autoencoder based on the encoding loss value until the encoding loss value is within a preset range, stopping adjusting the autoencoder, and using the encoder in the adjusted autoencoder as the encoder model.

[0006] In some embodiments of the present application, the adversarial network includes a generator and a discriminator, and the method for generating the generator model includes: inputting the training feature matrix into the generator to obtain a target generated image corresponding to the first modality image, calling the discriminator to identify the second modality image and the target generated image to obtain a first identification result, and adjusting the generator according to the first identification result to obtain the generator model.

[0007] In some embodiments of the present application, the generator includes a first generator and a second generator, and the inputting the training feature matrix into the generator to obtain a target generated image corresponding to the first modal image includes: inputting the training feature matrix into the first generator to obtain an initial generated image, and inputting the training feature matrix and the initial generated image into the second generator to obtain the target generated image, wherein the resolution of the target generated image is greater than the resolution of the initial generated image.

[0008] In some embodiments of the present application, the first generator includes an embedding layer and an upsampling layer, and the inputting the training feature matrix into the first generator to obtain the initial generated image includes: calling the embedding layer to map the training feature matrix to a preset dimensional space to obtain a training feature representation of the training feature matrix, and calling the upsampling layer to perform an upsampling operation on the training feature representation to obtain the initial generated image.

[0009] Through the above implementation, the encoder in the autoencoder is trained using the first modality image, so that the trained encoder model can accurately learn the image features of the first modality image. The adversarial network is trained by training the feature matrix and the second modality image. Since the first modality image and the second modality image have different imaging perspectives, the generator model in the trained adversarial model can learn the commonalities and differences between the first modality image and the second modality image, and understand the conversion relationship between images of different imaging perspectives, so that the generator model can effectively perform cross-modal conversion on images of different perspectives and expressions.

[0010] On the other hand, the present application provides a method for cross-modal conversion of images, the method comprising: acquiring a test modality image, inputting the test modality image into a cross-modal model, and obtaining a modality conversion image of the test modality image, wherein the test modality image and the modality conversion image have different imaging perspectives, and the cross-modal model comprises an encoder model and a generator model.

[0011] In some embodiments of the present application, generation of the cross-modal model includes: acquiring a first modality image and a second modality image corresponding to the first modality image, the first modality image and the second modality image having different imaging perspectives, using the first modality image to train an encoder in a preset autoencoder to obtain the encoder model, training a preset adversarial network based on the second modality image and a training feature matrix output by the encoder for the first modality image to obtain an adversarial model corresponding to the adversarial network, the adversarial model including the generator model.

[0012] In some embodiments of the present application, inputting the test modality image into the cross-modal model to obtain the modality conversion image of the test modality image includes: calling the encoder model to perform feature extraction on the test modality image to obtain a modality feature matrix, and inputting the modality feature matrix into the generator model to obtain the modality conversion image.

[0013] In some embodiments of the present application, the generator model includes a first generator model and a second generator model, and inputting the modal feature matrix into the generator model to obtain the modal conversion image includes: inputting the modal feature matrix into the first generator model to obtain an initial modal image, and inputting the modal feature matrix and the initial modal image into the second generator model to obtain the modal conversion image, wherein the resolution of the modal conversion image is greater than the resolution of the initial modal image.

[0014] In some embodiments of the present application, the first generator model includes an embedding layer and an upsampling layer, and the inputting the modal feature matrix into the first generator to obtain the initial modal image includes: calling the embedding layer to map the modal feature matrix to a preset dimensional space to obtain a modal feature representation of the modal feature matrix, and calling the upsampling layer to perform an upsampling operation on the modal feature representation to obtain the initial modal image.

[0015] Through the above implementation, since the cross-modal model has learned the specific method of cross-modal conversion of images with different perspectives and expressions, the cross-modal model can convert the content and perspective in the test modality image at the same time, thereby accurately realizing the cross-modal conversion of the image.

[0016] On the other hand, the present application provides an electronic device, comprising: a memory storing at least one instruction; and a processor executing the at least one instruction to implement the cross-modal model training method or the image cross-modal conversion method. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a structural diagram of an electronic device provided in one embodiment of the present application.

[0018] Figure 2 It is a flowchart of a cross-modal model training method provided in one embodiment of the present application.

[0019] Figure 3 It is a flowchart of a method for training an encoder model provided in one embodiment of the present application.

[0020] Figure 4 It is a flowchart of a training method for an adversarial model provided in one embodiment of the present application.

[0021] Figure 5 This is a flowchart of a training method for a generator model provided in one embodiment of the present application.

[0022] Figure 6 It is a flowchart of a method for cross-modal image conversion provided by an embodiment of the present application.

[0023] Figure 7 It is a flowchart of a method for generating a modality conversion image provided by an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0026] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0027] In the field of medical technology, the image information obtained by a single-modality imaging device often cannot fully reflect the complex characteristics of the disease. In order to accurately judge the condition, it is necessary to obtain medical images of multiple modalities to assist doctors in diagnosing the condition, for example, using magnetic resonance images and positron emission tomography images to assist in the diagnosis of Alzheimer's disease. In related technologies, the acquisition of medical images of different modalities requires a lot of time and financial resources, and it is impossible to convert the content in the medical images, making it difficult to achieve cross-modal conversion of images.

[0028] In order to solve the above technical problems, the present application provides a cross-modal model training method, an image cross-modal conversion method and an electronic device, which can effectively perform cross-modal conversion on images. The virtual human body model driving method provided in the embodiment of the present application can be applied to one or more electronic devices.

[0029] like Figure 1 1 is a block diagram of an electronic device provided in an embodiment of the present application. The electronic device 10 may be a mobile phone, a tablet computer, a laptop computer, or a computer. The present embodiment of the present application does not impose any restrictions on the specific type of the electronic device.

[0030] like Figure 1 As shown, the electronic device 10 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104 and a bus 105. The processor 103 is coupled to the communication module 101, the memory 102 and the input / output interface 104 through the bus 105.

[0031] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as universal serial bus (USB), controller area network bus (CAN, Controller Area Network). The wireless communication module may provide one or more wireless communication solutions such as wireless fidelity (Wi-Fi), Bluetooth (BT), mobile communication network, frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0032] The memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 103, and can be used to store executable programs (such as machine instructions) of other running programs, and can also be used to store user and application data. The random access memory may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0033] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110. The non-volatile memory may include a disk storage device and a flash memory.

[0034] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions, and when the multiple instructions are executed by the processor 103, the cross-modal model training method and the image cross-modal conversion method executed on the electronic device 10 can be implemented.

[0035] In other embodiments, Figure 1 The electronic device 10 shown also includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10 .

[0036] The processor 103 may include one or more processing units, for example, the processor 103 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), and / or a neural network processor (neural-network processing unit, NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0037] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute a computer program stored in the memory 102 to implement the above-mentioned cross-modal model training method and image cross-modal conversion method.

[0038] The input / output interface 104 is used to provide a channel for user input or output. For example, the input / output interface 104 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0039] The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the input / output interface 104 in the electronic device 10 .

[0040] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0041] like Figure 2 , which is a flow chart of a cross-modal model training method provided by an embodiment of the present application. According to different requirements, the order of each step in the flow chart can be adjusted according to the actual detection requirements, and some steps can be omitted. The execution subject of the method is an electronic device, such as Figure 1 An electronic device 1 is shown.

[0042] S11, acquiring a first modality image and a second modality image corresponding to the first modality image.

[0043] In at least one embodiment of the present application, the first modality image and the second modality image may be medical images obtained by imaging the same part (region) using different imaging techniques.

[0044] In at least one embodiment of the present application, the first modality image includes, but is not limited to: an optical imaging image, a computed tomography image (CT), an X-ray image, and a magnetic resonance imaging image (MR). When the first modality image is an optical image, the second modality image may be an optical coherence tomography image (OCT); or, when the first modality image is a magnetic resonance imaging image, the second modality image may be a positron emission tomography image (PET); or, when the first modality image is a computed tomography image, the second modality image may be a positron emission tomography image; or, when the first modality image is a computed tomography image, the second modality image may be a magnetic resonance imaging image; or, when the first modality image is an X-ray image, the second modality image may be a computed tomography image. In practical applications, the above examples are not limited thereto.

[0045] In some embodiments of the present application, the first modality image and the second modality image may be multiple images, and the electronic device may obtain multiple images from The Cancer Imaging Archive (TCIA), Radiological Society of North America (RSNA),

[0046] Multiple first modality images and second modality images corresponding to each first modality image are obtained from databases such as Digital Imaging and Communications in Medicine (DICOM), National Institutes of Health (NIH), and Open Access Data in the Public Domain (OADPD).

[0047] In this embodiment, the first modality image and the second modality image have different imaging perspectives. For example, when the first modality image is an optical image obtained by photographing the fundus of the eye with a fundus camera, the second modality image may be a cross-sectional (side) image obtained by scanning the side of the eye with an optical synchronous tomography scanner, wherein the eye cross-sectional image includes cross-sectional information of the eye.

[0048] In other embodiments of the present application, the electronic device may obtain multiple first modality images and the second modality image corresponding to each first modality image by other means, and the present application does not limit the method for obtaining the first modality images and the second modality images.

[0049] S12: Use the first modality image to train an encoder in a preset autoencoder to obtain an encoder model of the encoder.

[0050] In at least one embodiment of the present application, the autoencoder includes an encoder and a decoder. The encoder is used to extract features from the first modality image to obtain a training feature matrix, and the decoder is used to decode or reconstruct the training feature matrix.

[0051] In this embodiment, the encoder is trained using the first modality image, so that the trained encoder model can learn the image features of the first modality image.

[0052] In other embodiments of the present application, the autoencoder may also be replaced by other autoencoders, such as variational autoencoders (VAE), and the present application does not impose any limitation on this.

[0053] S13, training a preset adversarial network according to the second modality image and the training feature matrix output by the encoder for the first modality image, to obtain an adversarial model corresponding to the adversarial network.

[0054] In at least one embodiment of the present application, the adversarial network includes a generator and a discriminator, the generator includes a first generator and a second generator, the first generator and the second generator both include an embedding layer and an upsampling layer, and the network structures of the first generator and the second generator are basically the same.

[0055] In at least one embodiment of the present application, the training goal of the generator is to generate an image that is sufficiently similar to the second modality image so that the discriminator cannot distinguish between the second modality image and the image generated by the generator, and the training goal of the discriminator is to accurately distinguish (distinguish) the second modality image and the image generated by the generator, and the adversarial model also includes a generator model corresponding to the generator and a discriminator model corresponding to the discriminator.

[0056] In this embodiment, the adversarial network is trained through the training feature matrix of the first modality image and the second modality image, so that the trained generator model learns the correlation relationship between the first modality image and the second modality image, thereby being able to perform cross-modal conversion on the image.

[0057] S14, combines the encoder model and the generator model to obtain a cross-modal model.

[0058] In some embodiments of the present application, the electronic device may concatenate the encoder model with the generator model to obtain a cross-modal model.

[0059] Through the above implementation, the encoder in the autoencoder is trained using the first modality image, so that the trained encoder model can accurately learn the image features of the first modality image. The adversarial network is trained by training the feature matrix and the second modality image. Since the first modality image and the second modality image have different imaging perspectives, the generator model in the trained adversarial model can learn the commonalities and differences between the first modality image and the second modality image, and understand the conversion relationship between images of different imaging perspectives, so that the generator model can effectively perform cross-modal conversion on images of different perspectives and expressions.

[0060] In at least one embodiment of the present application, Figure 3 FIG. 1 is a flowchart of a method for training an encoder model provided in an embodiment of the present application, which specifically includes the following steps:

[0061] S121, calling the encoder to extract features from the first modality image to obtain a training feature matrix, and calling the decoder to reconstruct the training feature matrix to obtain a reconstructed image.

[0062] In some embodiments of the present application, the encoder includes one or more hidden layers. When the encoder includes multiple hidden layers, feature extraction of the first modality image includes: inputting the first modality image into the encoder to obtain a matrix output by each hidden layer, wherein the matrix output by each hidden layer is used as the input of the next hidden layer, and the electronic device uses the matrix output by the last hidden layer as a training feature matrix. Each hidden layer is composed of a cascade of a convolution layer, a batch normalization layer, and an activation function layer, and the activation function layer can be ReLu.

[0063] In some embodiments of the present application, the decoder may include a computing layer corresponding to each hidden layer, and the decoder and the encoder have a symmetrical structure. Each computing layer may be composed of a cascade of a deconvolution layer, a batch normalization layer, and an activation function layer, and the activation function layer may be ReLu. Since the decoder and the encoder have a symmetrical structure, the process of reconstructing the training feature matrix to obtain the reconstructed image is an inverse process to extracting features from the first modality image to obtain the training feature matrix.

[0064] In this embodiment, since the encoder extracts complete features of the first modality image, the encoder can generate a high-definition reconstructed image.

[0065] S122: Calculate a coding loss value of the autoencoder according to the first modality image and the reconstructed image, and adjust the autoencoder according to the coding loss value.

[0066] In some embodiments of the present application, the electronic device may determine a root mean square error between the first modality image and the reconstructed image as a coding loss value.

[0067] In other embodiments of the present application, the electronic device may further calculate a structural similarity index (SSIM) between the first modality image and the reconstructed image as a coding loss value, and the present application does not impose any limitation on the coding loss value.

[0068] In this embodiment, if the coding loss value is the root mean square error, the smaller the coding loss value, the more similar the reconstructed image is to the first modality image; or, if the coding loss value is the structural similarity index, the larger the coding loss value, the more similar the reconstructed image is to the first modality image.

[0069] S123, determining whether the encoding loss value is within a preset range.

[0070] In some embodiments of the present application, the preset range can be set voluntarily, and the present application does not limit this. For example, when the coding loss value is the root mean square error, in order to make the reconstructed image generated by the trained encoder model more similar to the first modal image, the value in the preset range can be set to be smaller. For example, the preset range can be [0.1, 0.2]. Alternatively, when the coding loss value is the structural similarity index, in order to make the reconstructed image generated by the trained encoder model more similar to the first modal image, the value in the preset range can be set to be larger. For example, the preset range can be [0.9, 0.95].

[0071] In this embodiment, when the encoding loss value is within the preset range, the electronic device determines that the autoencoder has converged, and executes step S124. Alternatively, when the encoding loss value is within the preset range, the electronic device determines that the autoencoder has not converged, and the process returns to step S121.

[0072] S124, stop adjusting the autoencoder, and use the encoder in the adjusted autoencoder as the encoder model.

[0073] In this embodiment, since the coding loss value is calculated by the reconstructed image and the first modality image, the encoder is trained by the coding loss value, so that the encoder can accurately learn the characteristics of the first modality image, thereby making the reconstructed image generated by the trained encoder model more accurate. In addition, since the preset range can be set according to the required similarity between the first modality image and the reconstructed image, whether the autoencoder converges can be determined by comparing the preset range and the coding loss value, which can improve the accuracy of the encoder model.

[0074] In at least one embodiment of the present application, the adversarial model includes a generator model and a discriminator model, such as Figure 4 FIG. 1 is a flowchart of a method for training an adversarial model provided in an embodiment of the present application, which specifically includes the following steps:

[0075] S131, input the training feature matrix into the generator to obtain a target generated image corresponding to the first modality image.

[0076] In some embodiments of the present application, the generator includes a first generator and a second generator, and the training feature matrix is ​​input into the generator to obtain a target generated image corresponding to the first modal image, including: the electronic device inputs the training feature matrix into the first generator to obtain an initial generated image, and inputs the training feature matrix and the initial generated image into the second generator to obtain a target generated image, wherein the resolution of the target generated image is greater than the resolution (resolution) of the initial generated image.

[0077] Specifically, the first generator includes an embedding layer and an upsampling layer. The electronic device inputs the training feature matrix into the first generator to obtain an initial generated image, including: the electronic device calls the embedding layer to map the training feature matrix to a preset dimensional space, obtains a training feature representation of the training feature matrix, and calls the upsampling layer to perform an upsampling operation on the training feature representation to obtain the initial generated image.

[0078] Specifically, the electronic device inputs the training feature matrix and the initial generated image into the second generator to obtain the target generated image, including: the electronic device downsamples the initial generated image to obtain a target matrix, fuses the training feature matrix with the target matrix to obtain a fused matrix, and performs an upsampling operation on the fused matrix to obtain the target generated image.

[0079] The preset dimensional space may be a low-dimensional space. For example, the preset dimensional space may be a two-dimensional space. The process of upsampling the fusion matrix to obtain the target generated image is substantially the same as the process of generating the initial generated image, so this application will not repeat the description.

[0080] In this embodiment, mapping the training feature matrix to a low-dimensional space is equivalent to reducing the dimension of the training feature matrix. Since the upsampling operation can improve the resolution, by upsampling the initial generated image, the resolution of the target generated image can be improved on the basis of the resolution of the initial generated image, so the resolution of the target generated image is greater than the resolution of the initial generated image. In addition, since the target generated image incorporates the information of the training feature matrix and the initial generated image, the target generated image can have richer image features.

[0081] S132, calling a discriminator to discriminate the second modality image and the target generated image to obtain a first discrimination result.

[0082] In some embodiments of the present application, the electronic device calls an identifier to identify the second modal image and the target generated image, and obtains a first identification result including: the electronic device extracts features from the second modal image and the target generated image, respectively, to obtain a first feature vector corresponding to the second modal image and a second feature vector corresponding to the target generated image; further, the electronic device classifies the first feature vector and the second feature vector, respectively, to obtain a first probability that the second modal image belongs to a preset category and a second probability that the target generated image belongs to a preset category, and determines the first probability and the second probability as the first identification result.

[0083] Among them, feature extraction of the second modal image and the target generated image includes operations such as convolution, pooling and full connection, and the electronic device can use a sigmoid function or a softmax function as a classifier to classify the first feature vector and the second feature vector. The preset category can be set by itself, and this application does not limit this. For example, the preset category can be true or false, so that the discriminator can distinguish the true from the false of the input image.

[0084] In this embodiment, since the training goal of the discriminator is to accurately identify (distinguish) the second modality image and the image generated by the generator, the first identification result can reflect the similarity between the second modality image and the target generated image.

[0085] S133, adjusting the generator according to the first identification result to obtain a generator model.

[0086] In some embodiments of the present application, the electronic device adjusts the parameters of the generator to obtain a generator model, where the parameters include, but are not limited to: learning rate, weight, bias, etc.

[0087] In this embodiment, since the first identification result can reflect the degree of similarity between the second modality image and the target generated image, adjusting the generator according to the first identification result can enable the generator model to generate an image that is sufficiently similar to the second modality image.

[0088] In other embodiments of the present application, the electronic device may also calculate a target loss value based on the target generated image and the second modality image, and adjust parameters of the generator based on the first identification result and the target loss value.

[0089] S134, using a cross entropy loss function to calculate the first identification result to obtain an identification loss value, and adjusting the discriminator according to the identification loss value to obtain a discriminator model.

[0090] In some embodiments of the present application, since the first identification result includes a first probability and a second probability, the electronic device can use a cross-entropy loss function to calculate the first probability and the second probability respectively, obtain a first loss value of the first probability and a second loss value of the second probability, and calculate the identification loss value based on the first loss value and the second loss value.

[0091] The method of calculating the identification loss value according to the first loss value and the second loss value can be set arbitrarily, and this application does not limit this. For example, the electronic device can determine the identification loss value by adding or weighting the first loss value and the second loss value.

[0092] In this embodiment, the identification loss value can be calculated multiple times, and the discriminator can be adjusted by each identification loss value to obtain an adjusted discriminator, until the identification loss value is within the preset configuration range, the adjustment is stopped, and the discriminator model is obtained. Among them, the preset configuration range can be set by oneself, and this application does not limit this.

[0093] In at least one embodiment of the present application, Figure 5 FIG. 1 is a flow chart of a method for training a generator model provided in an embodiment of the present application, which specifically includes the following steps:

[0094] S1341, determining whether the first identification result approaches a preset value.

[0095] In this embodiment, the preset value may be 0.5.

[0096] When the first probability and the second probability both approach 0.5, the electronic device executes step S1346; or, when the first probability or the second probability does not approach 0.5, the electronic device executes step S1342.

[0097] S1342, adjust the generator to obtain an adjusted generator.

[0098] In this embodiment, parameters such as the weight, bias, and learning rate of the generator may be adjusted, and this application does not impose any restrictions on this.

[0099] S1343, calling the adjusted generator to generate an updated generated image corresponding to the training feature matrix.

[0100] In this embodiment, the generation process of the updated generated image is substantially the same as the generation process of the target generated image, and thus will not be described again herein.

[0101] S1344, obtaining a second identification result of the adjusted discriminator on the updated generated image and the second modality image.

[0102] In some embodiments of the present application, the electronic device calls the adjusted discriminator to discriminate the updated image and the second modality image to obtain a second discrimination result, wherein the second discrimination result includes a third probability that the updated image belongs to a preset category and a fourth probability that the second modality image belongs to a preset category.

[0103] In this embodiment, the generation process of the updated generated image is substantially the same as the generation process of the target generated image, and the generation process of the second identification result is substantially the same as the generation process of the first identification result, so this application will not repeat the description.

[0104] S1345, determining whether the second identification result approaches a preset value.

[0105] If the third probability and the fourth probability approach 0.5, execute step S1346; or, if the third probability or the fourth probability does not approach 0.5, the process returns to step S1342 to continue adjusting the adjusted generator.

[0106] In some embodiments of the present application, the electronic device may also calculate an updated loss value based on the updated generated image and the second modality image, and adjust the parameters of the generator based on the second identification result and the updated loss value.

[0107] S1346, stop adjusting, and determine the adjusted generator as the generator model.

[0108] In at least one embodiment of the present application, Figure 6 , which is a flowchart of a method for image cross-modal conversion provided by an embodiment of the present application.

[0109] According to different requirements, the order of each step in the flowchart can be adjusted according to the actual detection requirements, and some steps can be omitted. The execution subject of the method is an electronic device, such as Figure 1 An electronic device 1 is shown.

[0110] S21, acquiring a test modality image.

[0111] In at least one embodiment of the present application, the method for acquiring the test modality image is substantially the same as the method for acquiring the first modality image and the second modality image, and thus the present application will not repeat the description.

[0112] S22, inputting the test modality image into the cross-modality model to obtain a modality conversion image of the test modality image.

[0113] In some embodiments of the present application, the test modality image and the modality conversion image have different imaging perspectives, and the cross-modal model includes an encoder model and a generator model. The generation of the cross-modal model can refer to the introduction of steps S11-S14, and this application will not repeat the description.

[0114] In some embodiments of the present application, Figure 7 FIG. 1 is a flow chart of a method for generating a modality conversion image provided by an embodiment of the present application. Specifically, the method comprises the following steps:

[0115] S221, calling the encoder model to extract features from the test modality image to obtain a modality feature matrix.

[0116] In some embodiments of the present application, the generation method of the modal feature matrix and the training feature matrix is ​​basically the same, so this application will not repeat the description.

[0117] S222, input the modal feature matrix into the generator model to obtain a modal conversion image.

[0118] In some embodiments of the present application, the generator model includes a first generator model and a second generator model, and the modal feature matrix is ​​input into the generator model to obtain a modal conversion image, including: the electronic device inputs the modal feature matrix into the first generator model to obtain an initial modal image, and inputs the modal feature matrix and the initial modal image into the second generator model to obtain a modal conversion image, wherein the resolution of the modal conversion image is greater than the resolution of the initial modal image.

[0119] Specifically, the first generator model includes an embedding layer and an upsampling layer. The electronic device inputs the modal feature matrix into the first generator to obtain an initial modal image, including: the electronic device calls the embedding layer to map the modal feature matrix to a preset dimensional space to obtain a modal feature representation of the modal feature matrix, and calls the upsampling layer to perform an upsampling operation on the modal feature representation to obtain the initial modal image.

[0120] The preset dimensional space may be a low-dimensional space. For example, the preset dimensional space may be a two-dimensional space. The generation method of the modal feature representation is substantially the same as the generation method of the training feature representation, and the generation method of the initial modal image is substantially the same as the generation method of the initial generated image, so this application will not repeat the description.

[0121] Through the above implementation, since the cross-modal model has learned the specific method of cross-modal conversion of images with different perspectives and expressions, the cross-modal model can convert the content and perspective in the test modality image at the same time, thereby accurately realizing the cross-modal conversion of the image.

[0122] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0123] The computer-readable storage medium may be an internal memory of the electronic device described in the above embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device.

[0124] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0125] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0127] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A cross-modal model training method, characterized in that: The method comprises: Acquire a first modality image and a second modality image corresponding to the first modality image; Using the first modality image to train an encoder in a preset autoencoder to obtain an encoder model of the encoder; Training a preset adversarial network according to the second modality image and the training feature matrix output by the encoder for the first modality image to obtain an adversarial model corresponding to the adversarial network, wherein the adversarial model includes a generator model; The encoder model and the generator model are combined to obtain a cross-modal model.

2. The cross-modal model training method according to claim 1, characterized in that: The autoencoder includes the encoder and the decoder, and the encoder in the preset autoencoder is trained using the first modality image to obtain an encoder model of the encoder including: Calling the encoder to perform feature extraction on the first modality image to obtain the training feature matrix, and calling the decoder to reconstruct the training feature matrix to obtain a reconstructed image; According to the first modality image and the reconstructed image, the coding loss value of the autoencoder is calculated, and the autoencoder is adjusted according to the coding loss value until the coding loss value is within a preset range, the adjustment of the autoencoder is stopped, and the encoder in the adjusted autoencoder is used as the encoder model.

3. The cross-modal model training method according to claim 1, characterized in that: The adversarial network includes a generator and a discriminator, and the method for generating the generator model includes: Inputting the training feature matrix into the generator to obtain a target generated image corresponding to the first modality image; Calling the discriminator to discriminate the second modal image and the target generated image to obtain a first discrimination result; The generator is adjusted according to the first identification result to obtain the generator model.

4. The cross-modal model training method according to claim 3, characterized in that: The generator includes a first generator and a second generator, and the step of inputting the training feature matrix into the generator to obtain a target generated image corresponding to the first modality image includes: Inputting the training feature matrix into the first generator to obtain an initial generated image; The training feature matrix and the initial generated image are input into the second generator to obtain the target generated image, wherein the resolution of the target generated image is greater than the resolution of the initial generated image.

5. The cross-modal model training method according to claim 4, characterized in that: The first generator includes an embedding layer and an upsampling layer, and the step of inputting the training feature matrix into the first generator to obtain an initial generated image includes: Calling the embedding layer to map the training feature matrix to a preset dimensional space to obtain a training feature representation of the training feature matrix; The upsampling layer is called to perform an upsampling operation on the training feature representation to obtain the initial generated image.

6. A method for image cross-modal conversion, characterized in that: The method comprises: Acquire test modality images; The test modality image is input into a cross-modality model to obtain a modality conversion image of the test modality image, wherein the cross-modality model includes an encoder model and a generator model.

7. The image cross-modal conversion method according to claim 6, characterized in that: The generation of the cross-modal model includes: Acquire a first modality image and a second modality image corresponding to the first modality image; Using the first modality image to train an encoder in a preset autoencoder to obtain the encoder model; A preset adversarial network is trained according to the second modality image and the training feature matrix output by the encoder for the first modality image to obtain an adversarial model corresponding to the adversarial network, wherein the adversarial model includes the generator model.

8. The image cross-modal conversion method according to claim 6, characterized in that: The step of inputting the test modality image into a cross-modality model to obtain a modality conversion image of the test modality image comprises: Calling the encoder model to perform feature extraction on the test modality image to obtain a modality feature matrix; The modal feature matrix is ​​input into the generator model to obtain the modal conversion image.

9. The image cross-modal conversion method according to claim 8, characterized in that: The generator model includes a first generator model and a second generator model, and the step of inputting the modal feature matrix into the generator model to obtain the modal conversion image includes: Inputting the modal feature matrix into the first generator model to obtain an initial modal image; The modal feature matrix and the initial modal image are input into the second generator model to obtain the modal conversion image, wherein the resolution of the modal conversion image is greater than the resolution of the initial modal image.

10. The image cross-modal conversion method according to claim 9, characterized in that: The first generator model includes an embedding layer and an upsampling layer, and the step of inputting the modality feature matrix into the first generator to obtain an initial modality image includes: Calling the embedding layer to map the modal feature matrix to a preset dimensional space to obtain a modal feature representation of the modal feature matrix; The upsampling layer is called to perform an upsampling operation on the modal feature representation to obtain the initial modal image.

11. An electronic device, characterized in that: The electronic device comprises: a memory storing at least one instruction; and A processor obtains instructions stored in the memory to implement the cross-modal model training method as described in any one of claims 1 to 5, or implements the image cross-modal conversion method as described in claims 6-10.

Images