Picture reconstruction method, system and device and storage medium

Abstract

The invention discloses a picture reconstruction method, system and device and a storage medium. The picture reconstruction method comprises the steps of extracting a text feature vector and an image feature vector corresponding to each picture in a picture training data set; fusing the text feature vector and the image feature vector through a first preset fusion method to obtain a first fused feature vector; based on the picture training data set and the first fusion feature vector, training a preset first initial picture reconstruction model to obtain a first picture reconstruction model; performing element-by-element multiplication on the text feature vector and the image feature vector to obtain a second fusion feature vector; and training the first picture reconstruction model based on the picture training data set and the second fusion feature vector to obtain a second picture reconstruction model, thereby improving the picture reconstruction accuracy of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of picture reconstruction, and in particular to a picture reconstruction method, system, device and storage medium. BACKGROUND

[0002] Low-resolution picture reconstruction in a complex environment is always a problem that cannot be ignored. At present, the main picture reconstruction method only learns the difference between high-resolution pictures and low-resolution pictures to accelerate the training process and improve accuracy, but residual learning cannot alleviate the training problem of deep network, and there will be problems of gradient disappearance or explosion in deep network, resulting in poor model training effect. SUMMARY

[0003] The present application aims to at least solve the technical problems existing in the prior art. To this end, the present application provides a picture reconstruction method, system, device and storage medium, which can use blurred pictures to provide global structure information and text description to supplement detailed semantic guidance, thereby improving the accuracy of picture reconstruction.

[0004] In a first aspect, the present application provides a picture reconstruction method, comprising the following steps:

[0005] Extracting a text feature vector and an image feature vector corresponding to each picture in a picture training data set;

[0006] Fusing the text feature vector and the image feature vector by a first preset fusion method to obtain a first fused feature vector;

[0007] Training a first initial picture reconstruction model based on the picture training data set and the first fused feature vector to obtain a first picture reconstruction model;

[0008] Element-wise multiplying the text feature vector and the image feature vector to obtain a second fused feature vector;

[0009] Training the first picture reconstruction model based on the picture training data set and the second fused feature vector to obtain a second picture reconstruction model.

[0010] According to the control method of the present application, at least the following beneficial effects are achieved:

[0011] The method extracts a text feature vector and an image feature vector corresponding to each picture in a picture training data set; fuses the text feature vector and the image feature vector by a first preset fusion method to obtain a first fusion feature vector; trains a preset first initial picture reconstruction model based on the picture training data set and the first fusion feature vector to obtain a first picture reconstruction model; element-wise multiplies the text feature vector and the image feature vector to obtain a second fusion feature vector; and trains the first picture reconstruction model based on the picture training data set and the second fusion feature vector to obtain a second picture reconstruction model. The application provides global structure information by fusing text features and picture features, supplements detailed semantic guidance by text description, and improves the accuracy of picture reconstruction by the model through twice feature fusion and model training.

[0012] According to some embodiments of the application, the first fusion feature vector is calculated by the following formula:

[0013] F {fusion} =α*F {image} +(1-α)*F {text}

[0014] Wherein, F {fusion} is the first fusion feature vector, F {image} is the image feature vector, F {text} is the text feature vector, and α is a preset image clarity confidence constant.

[0015] According to some embodiments of the application, the training process of the first picture reconstruction model includes:

[0016] The first loss value of the picture training data set in the first initial picture reconstruction model is calculated by the following formula:

[0017]

[0018] Wherein, L1 is the first loss value, p i is the prediction result of the i-th picture in the picture training data set, y i is the true result of the i-th picture in the picture training data set, γ is a first preset parameter, β is a second preset parameter, and N is the total number of pictures in the picture training data set.

[0019] The first initial picture reconstruction model is updated according to the first loss value until the loss value meets a first preset constraint condition, and the first picture reconstruction model is obtained.

[0020] According to some embodiments of the application, the training process of the second picture reconstruction model includes:

[0021] The second loss value of the picture training data set in the first picture reconstruction model is calculated by the following formula:

[0022]

[0023] wherein, L D is the first loss value, D(G(z)) is the prediction result of the picture training data set, D(x) is the true result of the picture training data set, and N is the total number of pictures in the picture training data set;

[0024] The first picture reconstruction model is updated according to the second loss value until the loss value meets a second preset constraint condition, and the second picture reconstruction model is obtained.

[0025] According to some embodiments of the present application, the picture reconstruction method further comprises:

[0026] Obtaining the text description and the training picture reconstruction result of the picture training data set in the second picture reconstruction model;

[0027] Calculating a comparison loss value based on the training picture reconstruction result and the text description;

[0028] The second picture reconstruction model is updated according to the comparison loss value until the comparison loss value meets a third preset constraint condition, and a third picture reconstruction model is obtained.

[0029] According to some embodiments of the present application, the picture reconstruction method further comprises:

[0030] Obtaining a picture to be reconstructed and a text description thereof;

[0031] Inputting the picture to be reconstructed and the text description thereof into the third picture reconstruction model to obtain a picture reconstruction result.

[0032] According to some embodiments of the present application, the comparison loss value is calculated by the following formula:

[0033]

[0034] wherein, L2 is the comparison loss value, I i is the text description of the i-th picture in the picture training data set, T j is the picture reconstruction result of the j-th picture in the picture training data set, I j is the text description of the j-th picture in the picture training data set, T i is the picture reconstruction result of the i-th picture in the picture training data set, τ is a temperature parameter, and sim() is a cosine similarity calculation function.

[0035] In a second aspect of the present application, a picture reconstruction system is provided, and the picture reconstruction system comprises:

[0036] a feature extraction module configured to extract a text feature vector and an image feature vector corresponding to each picture in a picture training data set;

[0037] a first feature fusion module configured to fuse the text feature vector and the image feature vector by using a first preset fusion method to obtain a first fusion feature vector;

[0038] a first model training module configured to train a first initial picture reconstruction model based on the picture training data set and the first fusion feature vector to obtain a first picture reconstruction model;

[0039] a second feature fusion module configured to multiply the text feature vector and the image feature vector element by element to obtain a second fusion feature vector;

[0040] a second model training module configured to train the first picture reconstruction model based on the picture training data set and the second fusion feature vector to obtain a second picture reconstruction model.

[0041] The system extracts a text feature vector and an image feature vector corresponding to each picture in a picture training data set, fuses the text feature vector and the image feature vector by using a first preset fusion method to obtain a first fusion feature vector, trains a first initial picture reconstruction model based on the picture training data set and the first fusion feature vector to obtain a first picture reconstruction model, multiplies the text feature vector and the image feature vector element by element to obtain a second fusion feature vector, and trains the first picture reconstruction model based on the picture training data set and the second fusion feature vector to obtain a second picture reconstruction model. The present application provides global structural information by fusing a text feature and a picture feature, supplements detailed semantic guidance by a text description, and improves the accuracy of picture reconstruction by the model through twice feature fusion and training of the model.

[0042] In a third aspect of the present application, a picture reconstruction electronic device is provided, and the picture reconstruction electronic device comprises at least one control processor and a memory connected to the at least one control processor in communication; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the picture reconstruction method described above.

[0043] In a fourth aspect of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores computer executable instructions, and the computer executable instructions are used to enable a computer to execute the picture reconstruction method described above.

[0044] It should be noted that the beneficial effects between the second aspect to the fourth aspect of the present application and the prior art are the same as the beneficial effects between the above-mentioned picture reconstruction system and the prior art, which will not be described here.

[0045] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0046] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the appended drawings.

[0047] Figure 1 is a flow chart of a picture reconstruction method according to an embodiment of the present application;

[0048] Figure 2 is a structural schematic diagram of an embodiment of a picture reconstruction system provided by the present application;

[0049] Figure 3 is a structural schematic diagram of an embodiment of an electronic device provided by the present application. DETAILED DESCRIPTION

[0050] The embodiments of the present application are described in detail below with reference to the accompanying drawings. The embodiments described below are examples for explaining the present application and should not be understood as limiting the present application.

[0051] In the description of the present application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and should not be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of the indicated technical features.

[0052] In the description of the present application, it should be understood that the orientation description, such as up, down, etc., indicates the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and should not be understood as indicating or implying that the device or element indicated must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present application.

[0053] In the description of the present application, it should be noted that the words such as setting, installing, connecting, etc. should be understood in a broad sense unless otherwise explicitly limited, and the person skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0054] Low-resolution picture reconstruction in complex environment is always a problem that cannot be ignored. At present, the main picture reconstruction method only learns the difference between high-resolution pictures and low-resolution pictures to accelerate the training process and improve the accuracy, but the residual learning cannot alleviate the training problem of deep network, and there will be problems of gradient disappearance or explosion in deep network, resulting in poor model training effect.

[0055] In order to solve the above technical defects, the embodiment of the application provides a picture reconstruction method, system, device and storage medium.

[0056] Please refer to Figure 1 It is a flowchart of a picture reconstruction method provided by the embodiment of the application, which is applied to an electronic device, which can be a server and the like. As shown in Figure 1 The picture reconstruction method comprises the following steps.

[0057] Step S101, extracting the text feature vector and the image feature vector corresponding to each picture in the picture training data set;

[0058] Step S102, fusing the text feature vector and the image feature vector by a first preset fusion method to obtain a first fused feature vector;

[0059] Step S103, training a preset first initial picture reconstruction model based on the picture training data set and the first fused feature vector to obtain a first picture reconstruction model;

[0060] Step S104, multiplying the text feature vector and the image feature vector element by element to obtain a second fused feature vector;

[0061] Step S105, training the first picture reconstruction model based on the picture training data set and the second fused feature vector to obtain a second picture reconstruction model.

[0062] Specifically, the above-mentioned extraction of the text feature vector can be an input text description (such as "a hawk standing on the top of the snow-capped mountain, with clear feather details"), and the text feature vector is obtained by pre-training a language model and a learnable prompt vector.

[0063] Specifically, the above-mentioned extraction of the image feature vector can be a multi-scale image feature map obtained by adapting the fuzzy input through a lightweight CNN (such as ResNet34) and a deformable convolution (Deformable Convolution).

[0064] The method extracts a text feature vector and an image feature vector corresponding to each picture in a picture training data set; fuses the text feature vector and the image feature vector by a first preset fusion method to obtain a first fusion feature vector; trains a preset first initial picture reconstruction model based on the picture training data set and the first fusion feature vector to obtain a first picture reconstruction model; element-wise multiplies the text feature vector and the image feature vector to obtain a second fusion feature vector; and trains the first picture reconstruction model based on the picture training data set and the second fusion feature vector to obtain a second picture reconstruction model. The application provides global structure information by fusing text features and picture features, supplements detailed semantic guidance by text description, and improves the accuracy of picture reconstruction by the model through twice feature fusion and model training.

[0065] In some embodiments, the first fusion feature vector is calculated by the following formula:

[0066] F {fusion} = a * F {image} + (1-a) * F {text}

[0067] wherein, F {fusion} is the first fusion feature vector, F {image} is the image feature vector, F {text} is the text feature vector, and a is a preset image clarity confidence constant.

[0068] The application extracts features of key regions in the picture by combining the prompt word information of the user, and lays a foundation for positioning and reconstruction of key information in the blurred picture through multi-modal feature fusion.

[0069] In some embodiments, the training process of the first picture reconstruction model includes:

[0070] Step S201, calculating a first loss value of the picture training data set in the first initial picture reconstruction model by the following formula:

[0071]

[0072] wherein, L1 is the first loss value, p i is a prediction result of the i-th picture in the picture training data set, y i is a true result of the i-th picture in the picture training data set, γ is a first preset parameter, β is a second preset parameter, and N is a total number of pictures in the picture training data set;

[0073] Step S202, updating the first initial picture reconstruction model according to the first loss value until the loss value meets a first preset constraint condition to obtain the first picture reconstruction model.

[0074] Specifically, in some embodiments, the first picture reconstruction model comprises:

[0075] Down-sampling branch: extract context features.

[0076] Up-sampling branch: integrate text-guided skip connection.

[0077] Residual Dense Block (RRDB): enhance the ability of detail recovery.

[0078] Residual Connections:

[0079] The introduction of residual connections can alleviate the problem of gradient vanishing in deep networks, making it possible to train deeper networks. The form of the residual block is as follows:

[0080] H(x) = F(x) + x;

[0081] where H(x) is the output after the residual block, F(x) is the residual mapping, and x is the input feature map directly passed to the output.

[0082] Dense Connections:

[0083] The dense connection mechanism allows each layer to take the feature maps of all previous layers as input, which can promote feature reuse and reduce the number of parameters.

[0084] Attention Mechanisms:

[0085] Attention mechanisms can help the model focus on more important areas.

[0086] In some embodiments, the training process of the second picture reconstruction model comprises:

[0087] Step S301, calculate the second loss value of the picture training data set in the first picture reconstruction model by the following formula:

[0088]

[0089] where L D is the first loss value, D(G(z)) is the predicted result of the picture training data set, D(x) is the true result of the picture training data set, and N is the total number of pictures in the picture training data set;

[0090] Step S302, update the first picture reconstruction model according to the second loss value until the loss value meets the second preset constraint condition, and obtain the second picture reconstruction model.

[0091] Specifically, the second picture reconstruction model includes a multi-scale discriminator, and specifically further includes:

[0092] 3 PatchGANs (input sizes are 256x256, 512x512 and 1024x1024 respectively), which respectively process image features of different resolutions.

[0093] Image processing: according to image features F of different input sizes {image} , the generated image G(z) is scaled to three different scales:

[0094] (G(z)1=resize(F {image} ,256x256))

[0095] (G(z)2=resize(F {image} ,512x512))

[0096] (G(z)3=resize(F {image} ,1024x1024))

[0097] wherein the real image is also scaled to the corresponding size. The text embedding vector is taken as input and mapped to a dimension matching the image feature through a fully connected layer.

[0098] Text processing: the text is encoded into a vector by a pre-trained language model (such as BERT). The text vector is mapped to a dimension aligned with the image feature space through a fully connected layer. Feature fusion: at the middle layer of the discriminator (such as after the convolutional layer), the text feature is fused with the image feature, and the fusion method can be:

[0099] [F fused =Concat(F {image} ,t proj ) or F fused =F {image} ⊙t proj ]

[0100] wherein (⊙) represents element-wise multiplication, F {image} is the image feature, t proj is the text feature, and F fused is the fused feature.

[0101] The present application realizes image detail reconstruction optimization through progressive generation and pit generation by combining the dynamic features of multiple modalities.

[0102] In some embodiments, the picture reconstruction method further includes:

[0103] Step S401, obtaining a text description and a training picture reconstruction result of a picture training data set in a second picture reconstruction model;

[0104] Step S402, calculate a contrast loss value based on the training picture reconstruction result and the text description;

[0105] Step S403, update the second picture reconstruction model according to the contrast loss value until the contrast loss value meets a third preset constraint condition, and obtain a third picture reconstruction model.

[0106] The present application provides global structure information through a blurred image, and text description supplements detailed semantic guidance, realizes dual goals of "image inpainting + semantic enhancement", and improves the accuracy of picture reconstruction.

[0107] In some embodiments, the picture reconstruction method further comprises:

[0108] Step S501, obtain a picture to be reconstructed and a text description thereof;

[0109] Step S502, input the picture to be reconstructed and the text description thereof into the third picture reconstruction model to obtain a picture reconstruction result.

[0110] The present application uses a cascade network structure through a progressive generation strategy, gradually refines from low resolution to high resolution, and introduces text semantic guidance at each stage, thereby improving the accuracy of picture reconstruction.

[0111] In some embodiments, the contrast loss value is calculated by the following formula:

[0112]

[0113] Wherein, L2 is the contrast loss value, I i is the text description of the i-th picture in the picture training data set, T j is the picture reconstruction result of the j-th picture in the picture training data set, I j is the text description of the j-th picture in the picture training data set, T i is the picture reconstruction result of the i-th picture in the picture training data set, τ is a temperature parameter, and sim() is a cosine similarity calculation function.

[0114] The present application maximizes the similarity between the matched image and text pairs, and minimizes the similarity between the unmatched pairs, thereby improving the accuracy of the picture reconstruction result.

[0115] Specifically, in order to facilitate the understanding of those skilled in the art, a set of best embodiments is provided as follows:

[0116] I. Data acquisition:

[0117] Obtain a picture to be reconstructed and a text description thereof;

[0118] Extract the text feature vector and the image feature vector corresponding to each picture in the picture training data set.

[0119] II. First picture reconstruction model training

[0120] Fuse the text feature vector and the image feature vector by the first preset fusion method to obtain a first fusion feature vector. Specifically, the first fusion feature vector is calculated by the following formula:

[0121] F {fusion} = a * F {image} + (1-a) * F {text}

[0122] Wherein, F {fusion} is the first fusion feature vector, F {image} is the image feature vector, F {text} is the text feature vector, and a is a preset image clarity confidence constant.

[0123] Based on the picture training data set and the first fusion feature vector, the first initial picture reconstruction model is trained to obtain the first picture reconstruction model, wherein the training process of the first picture reconstruction model comprises:

[0124] The first loss value of the picture training data set in the first initial picture reconstruction model is calculated by the following formula:

[0125]

[0126] Wherein, L1 is the first loss value, p i is the prediction result of the i-th picture in the picture training data set, y i is the true result of the i-th picture in the picture training data set, γ is a first preset parameter, β is a second preset parameter, and N is the total number of pictures in the picture training data set.

[0127] The first initial picture reconstruction model is updated according to the first loss value until the loss value meets the first preset constraint condition, and the first picture reconstruction model is obtained.

[0128] III. Second picture reconstruction model training

[0129] Element-wise multiply the text feature vector and the image feature vector to obtain a second fusion feature vector;

[0130] Based on the picture training data set and the second fusion feature vector, the first picture reconstruction model is trained to obtain the second picture reconstruction model, wherein the training process of the second picture reconstruction model comprises:

[0131] The second loss value of the picture training data set in the first picture reconstruction model is calculated by the following formula:

[0132]

[0133] wherein, L D is the first loss value, D(G(z)) is the prediction result of the picture training data set, D(x) is the true result of the picture training data set, and N is the total number of pictures in the picture training data set;

[0134] updating the first picture reconstruction model according to the second loss value until the loss value meets the second preset constraint condition, to obtain a second picture reconstruction model.

[0135] Four, third picture reconstruction model training:

[0136] obtaining the text description and the training picture reconstruction result of the picture training data set in the second picture reconstruction model;

[0137] calculating the contrast loss value based on the training picture reconstruction result and the text description according to the following formula:

[0138]

[0139] wherein, L2 is the contrast loss value, I i is the text description of the i-th picture in the picture training data set, T j is the picture reconstruction result of the j-th picture in the picture training data set, I j is the text description of the j-th picture in the picture training data set, T i is the picture reconstruction result of the i-th picture in the picture training data set, and τ is a temperature parameter. sim() is a cosine similarity calculation function.

[0140] updating the second picture reconstruction model according to the contrast loss value until the contrast loss value meets the third preset constraint condition, to obtain a third picture reconstruction model.

[0141] Four, reconstruction result output:

[0142] obtaining a picture to be reconstructed and its text description;

[0143] inputting the picture to be reconstructed and its text description into the third picture reconstruction model to obtain a picture reconstruction result.

[0144] In addition, with reference to Figure 2 , one embodiment of the present application provides a picture reconstruction system, comprising a feature extraction module 1100, a first feature fusion module 1200, a first model training module 1300, a second feature fusion module 1400, and a second model training module 1500, wherein:

[0145] The feature extraction module 1100 is configured to extract the text feature vector and the image feature vector corresponding to each picture in the picture training data set;

[0146] The first feature fusion module 1200 is configured to fuse the text feature vector and the image feature vector by using a first preset fusion method to obtain a first fusion feature vector;

[0147] The first model training module 1300 is configured to train a preset first initial picture reconstruction model based on the picture training data set and the first fusion feature vector to obtain a first picture reconstruction model;

[0148] The second feature fusion module 1400 is configured to multiply the text feature vector and the image feature vector element by element to obtain a second fusion feature vector;

[0149] The second model training module 1500 is configured to train the first picture reconstruction model based on the picture training data set and the second fusion feature vector to obtain a second picture reconstruction model.

[0150] The system extracts the text feature vector and the image feature vector corresponding to each picture in the picture training data set, fuses the text feature vector and the image feature vector by using a first preset fusion method to obtain a first fusion feature vector, trains a preset first initial picture reconstruction model based on the picture training data set and the first fusion feature vector to obtain a first picture reconstruction model, multiplies the text feature vector and the image feature vector element by element to obtain a second fusion feature vector, and trains the first picture reconstruction model based on the picture training data set and the second fusion feature vector to obtain a second picture reconstruction model. The application provides global structure information by fusing the text feature and the picture feature, supplements the detailed semantic guidance by the text description, and improves the accuracy of the picture reconstruction model by twice feature fusion and model training.

[0151] It should be noted that the system embodiment and the method embodiment described above are based on the same inventive concept, and therefore the related content of the method embodiment described above is also applicable to the system embodiment, which will not be described here again.

[0152] Figure 3 A hardware structure diagram of picture reconstruction provided by an embodiment of the application is shown.

[0153] The picture reconstruction device can include a processor 301 and a memory 302 having computer program instructions stored therein.

[0154] In particular, the processor 301 can include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or can be configured to implement one or more integrated circuits that embody the embodiments of the present application.

[0155] The memory 302 can include mass storage for data or instructions. As an example and not by way of limitation, the memory 302 can include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc (e.g., a compact disc (CD) or a digital versatile disc (DVD)), a solid-state drive (SSD), a USB drive, or a combination of two or more of these. Where appropriate, the memory 302 can include removable or non-removable (or fixed) media, where appropriate. Where appropriate, the memory 302 can be internal or external to the integrated gateway disaster recovery device. In particular embodiments, the memory 302 is non-volatile, solid-state memory.

[0156] In some embodiments, the memory 302 can include read-only memory (ROM), random-access memory (RAM), a disk storage medium, an optical storage medium, a flash memory device, electrical, optical, or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software that, when executed (by one or more processors), is operable to perform operations described with reference to the methods according to an aspect of the present disclosure.

[0157] The processor 301 implements any one of the picture reconstruction methods in the above embodiments by reading and executing computer program instructions stored in the memory 302.

[0158] In one example, the picture reconstruction device can also include a communication interface 303 and a bus 310. As shown, the processor 301, the memory 302, and the communication interface 303 are connected through the bus 310 and complete communication with each other. Figure 3

[0159] The communication interface 303 is mainly used to realize the communication between the modules, devices, units and / or equipment in the embodiments of the present application.

[0160] ​Bus 310 includes a hardware, software, or both that couples components of the picture reconstruction device to each other. As an example and not by way of limitation, the bus can include an accelerated graphics port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a low pin count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards board (VLB) bus, or another suitable bus or a combination of two or more of these. Where appropriate, bus 310 can include one or more buses. Although this application describes and shows a particular bus, this application contemplates any suitable bus or interconnect.

[0161] The picture reconstruction device can perform the picture reconstruction method in this application based on the three-dimensional design model, thereby realizing the picture reconstruction method and system described in this application. Figure 1 and Figure 2 The picture reconstruction method and system described in this application.

[0162] In addition, in combination with the picture reconstruction method in the above embodiments, this application can provide a computer storage medium to realize. The computer storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to realize any one of the picture reconstruction methods in the above embodiments.

[0163] It needs to be clear that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after understanding the spirit of this application.

[0164] The functional blocks shown in the structural block diagrams above can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, and the like. When implemented in software, the elements of the present application are program or code segments that are used to perform the required tasks. The program or code segments can be stored in a machine-readable medium, or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. A "machine-readable medium" includes any medium that can store or transport information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. The code segments can be downloaded via computer networks such as the Internet, intranets, and the like.

[0165] It is also important to note that the examples mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the examples, or in an order different from the examples, or several steps can be performed simultaneously.

[0166] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other processing device to operate in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks. The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other processing device to operate in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0167] The above merely describes a specific implementation of the present application. Those skilled in the art can clearly understand the specific working processes of the system, modules and units described above for the convenience and brevity of description, and can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein again. It should be understood that the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A method for image reconstruction, characterized in that: The image reconstruction method comprises: Extract the text feature vector and image feature vector corresponding to each picture in the picture training dataset; fusing the text feature vector and the image feature vector using a first preset fusion method to obtain a first fused feature vector; Based on the image training data set and the first fused feature vector, a preset first initial image reconstruction model is trained to obtain a first image reconstruction model; Multiplying the text feature vector and the image feature vector element by element to obtain a second fused feature vector; Based on the image training data set and the second fused feature vector, the first image reconstruction model is trained to obtain a second image reconstruction model.

2. The image reconstruction method according to claim 1, wherein: The first fused feature vector is calculated by the following formula: F {fusion} =α*F {image} +(1-a)*F {text} Among them, F {fusion} is the first fusion feature vector, F {image} is the image feature vector, F {text} is the text feature vector, and α is the preset image clarity confidence constant.

3. The image reconstruction method according to claim 1, wherein: The training process of the first image reconstruction model includes: The first loss value of the image training dataset in the first initial image reconstruction model is calculated by the following formula: Among them, L1 is the first loss value, p i is the prediction result of the i-th picture in the picture training dataset, y i is the true result of the i-th image in the image training dataset, γ is the first preset parameter, β is the second preset parameter, and N is the total number of images in the image training dataset; The first initial image reconstruction model is updated according to the first loss value until the loss value satisfies a first preset constraint condition, thereby obtaining the first image reconstruction model.

4. The image reconstruction method according to claim 1, wherein: The training process of the second image reconstruction model includes: The second loss value of the image training dataset in the first image reconstruction model is calculated by the following formula: Among them, L D is the first loss value, D(G(z)) is the predicted result of the image training dataset, D(x) is the actual result of the image training dataset, and N is the total number of images in the image training dataset; The first image reconstruction model is updated according to the second loss value until the loss value satisfies a second preset constraint condition, thereby obtaining the second image reconstruction model.

5. The image reconstruction method according to claim 4, characterized in that: The image reconstruction method further includes: Obtaining, in the second image reconstruction model, a text description of the image training dataset and a training image reconstruction result; Calculating a contrast loss value based on the training image reconstruction result and the text description; The second image reconstruction model is updated according to the contrast loss value until the contrast loss value satisfies a third preset constraint condition, thereby obtaining a third image reconstruction model.

6. The image reconstruction method according to claim 5, characterized in that: The image reconstruction method further includes: Get the image to be reconstructed and its text description; The image to be reconstructed and its text description are input into the third image reconstruction model to obtain an image reconstruction result.

7. The image reconstruction method according to claim 5, characterized in that: The contrast loss value is calculated by the following formula: Among them, L2 is the contrast loss value, I i is the text description of the i-th picture in the picture training dataset, T j is the image reconstruction result of the jth image in the image training dataset, I j is the text description of the jth image in the image training dataset, T i is the image reconstruction result of the i-th image in the image training dataset, τ is the temperature parameter, and sim() is the cosine similarity calculation function.

8. An image reconstruction system, characterized in that: The image reconstruction system comprises: Feature extraction module, used to extract text feature vectors and image feature vectors corresponding to each picture in the picture training dataset; A first feature fusion module, configured to fuse the text feature vector and the image feature vector using a first preset fusion method to obtain a first fused feature vector; A first model training module is configured to train a preset first initial image reconstruction model based on the image training data set and the first fused feature vector to obtain a first image reconstruction model; A second feature fusion module, configured to perform element-by-element multiplication of the text feature vector and the image feature vector to obtain a second fused feature vector; The second model training module is used to train the first image reconstruction model based on the image training data set and the second fused feature vector to obtain a second image reconstruction model.

9. An image reconstruction device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform an image reconstruction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the image reconstruction method according to any one of claims 1 to 7.

Images