Image reconstruction method, image reconstruction device, electronic device and storage medium

Abstract

The image reconstruction method, image reconstruction device, electronic device and storage medium provided by the present application relate to the field of image processing technology. In the present application, firstly, a first target image and a second target image are obtained, wherein the first target image has a first target object, the second target image has a second target object, and the first target object and the second target object have different type information; secondly, object features are extracted from the first target image and the second target image to obtain object feature information corresponding to the first target object and the second target object; then, image reconstruction is performed based on the object feature information to obtain a target reconstructed image, wherein the target reconstructed image has a second target object, and in the target reconstructed image, the second target object has the posture information of the first target object. Based on the above method, the scope of application of image reconstruction can be expanded, and the problem of poor reconstruction effect existing in the process of reconstructing images in the prior art can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image reconstruction method, an image reconstruction device, an electronic device and a storage medium. Background Art

[0002] Image reconstruction generally refers to the technology of reconstructing the corresponding image based on the feature information obtained by feature extraction of the object in the image. For example, in some applications, the features (such as expression, etc.) in the image of user A can be fused into the image of user B to obtain a new image, and the new image can have user B, and user B also has the features of user A in the corresponding image, such as user B has the expression of user A in the new image.

[0003] However, the inventors have discovered that, in the prior art, the above-mentioned image reconstruction technology is still limited to feature fusion between people to achieve corresponding image reconstruction, that is, it is limited to image objects of the same type, which greatly limits the scope of application of image reconstruction and results in poor image reconstruction effect. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide an image reconstruction method, an image reconstruction device, an electronic device and a storage medium to expand the scope of application of image reconstruction and improve the problem of poor reconstruction effect in the process of image reconstruction in the prior art.

[0005] To achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0006] An image reconstruction method, comprising:

[0007] Acquire a first target image and a second target image respectively, wherein the first target image has a first target object, the second target image has a second target object, and the first target object and the second target object have different type information;

[0008] Extracting object features from the first target image and the second target image to obtain object feature information corresponding to the first target object and the second target object;

[0009] An image is reconstructed based on the object feature information to obtain a target reconstructed image, wherein the target reconstructed image has the second target object, and in the target reconstructed image the second target object has the posture information of the first target object.

[0010] In a preferred embodiment of the present application, in the above-mentioned image reconstruction method, the step of extracting object features from the first target image and the second target image to obtain object feature information corresponding to the first target object and the second target object includes:

[0011] Acquire a first sample image and a second sample image, wherein the object in the first sample image has the same type information as the first target object, and the object in the second sample image has the same type information as the second target object;

[0012] An image reconstruction model is obtained by training based on the first sample image and the second sample image, wherein the image reconstruction model is a neural network model;

[0013] The first target image and the second target image are processed based on the image reconstruction model to obtain object feature information corresponding to the first target object and the second target object.

[0014] In a preferred embodiment of the present application, in the above-mentioned image reconstruction method, the step of obtaining an image reconstruction model based on the first sample image and the second sample image training includes:

[0015] Build the target neural network model;

[0016] Performing image reconstruction on the first sample image and the second sample image based on the target neural network model to obtain a first reconstructed image to be processed, and performing image reconstruction on the acquired third sample image and the first reconstructed image to be processed based on the target neural network model to obtain a second reconstructed image to be processed, wherein the object in the third sample image is the same as the object in the second sample image;

[0017] The corresponding image reconstruction loss is calculated based on the third sample image and the second reconstructed image to be processed, and the network parameters of the target neural network model are updated based on the image reconstruction loss to obtain an image reconstruction model.

[0018] In a preferred selection of the embodiment of the present application, in the above-mentioned image reconstruction method, the step of performing image reconstruction on the first sample image and the second sample image based on the target neural network model to obtain a first reconstructed image to be processed, and performing image reconstruction on the acquired third sample image and the first reconstructed image to be processed based on the target neural network model to obtain a second reconstructed image to be processed includes:

[0019] Performing key point detection on the first target image based on a first detection network included in the target neural network model to obtain first key point information corresponding to the first target image, and performing key point detection on the second target image based on a second detection network included in the target neural network model to obtain second key point information corresponding to the second target image;

[0020] Processing the first key point information and the second key point information based on the dense motion network included in the target neural network model to obtain corresponding dense optical flow information and occlusion map information, and performing feature encoding on the second target image based on the feature encoding network included in the target neural network model to obtain corresponding feature vector information;

[0021] Performing a target vector processing operation based on the dense optical flow information, the occlusion map information, and the feature vector information to obtain corresponding target feature vector information, and performing image reconstruction on the target feature vector information based on an image reconstruction network included in the target neural network model to obtain a corresponding first reconstructed image to be processed;

[0022] Based on the second detection network, key point detection is performed on the acquired third sample image and the first to-be-processed reconstructed image respectively to obtain third key point information corresponding to the third sample image and fourth key point information corresponding to the first to-be-processed reconstructed image, and the third key point information and the fourth key point information are processed based on the dense motion network to obtain corresponding new dense optical flow information and new occlusion map information;

[0023] Based on the feature encoding network, feature encoding is performed on the first to-be-processed reconstructed image to obtain corresponding new feature vector information, and a target vector processing operation is performed based on the new dense optical flow information, the new occlusion map information and the new feature vector information to obtain corresponding new target feature vector information;

[0024] Image reconstruction is performed on the new target feature vector information based on the image reconstruction network to obtain a corresponding second reconstructed image to be processed.

[0025] In a preferred embodiment of the present application, in the above-mentioned image reconstruction method, the step of obtaining an image reconstruction model based on the first sample image and the second sample image training further includes:

[0026] After obtaining the third key point information corresponding to the third sample image, calculating the corresponding equivariance loss based on the third key point information;

[0027] The network parameters of the second detection network are updated based on the equivariant loss.

[0028] In a preferred embodiment of the present application, in the above-mentioned image reconstruction method, the step of processing the first target image and the second target image based on the image reconstruction model to obtain object feature information corresponding to the first target object and the second target object includes:

[0029] Performing key point detection on the first target image based on a first detection network included in the image reconstruction model to obtain first key point information corresponding to the first target image, and performing key point detection on the second target image based on a second detection network included in the image reconstruction model to obtain second key point information corresponding to the second target image;

[0030] Processing the first key point information and the second key point information based on a dense motion network included in the image reconstruction model to obtain corresponding dense optical flow information and occlusion map information, and performing feature encoding on the second target image based on a feature encoding network included in the image reconstruction model to obtain corresponding feature vector information;

[0031] A target vector processing operation is performed based on the dense optical flow information, the occlusion map information and the feature vector information to obtain corresponding target feature vector information, and the target feature vector information is used as object feature information corresponding to the first target object and the second target object.

[0032] In a preferred embodiment of the present application, in the above-mentioned image reconstruction method, the first target image is a plurality of frames, and each frame of the first target image is reconstructed with the second target image to obtain a corresponding frame of target reconstructed image, and the image reconstruction method further includes:

[0033] Determining a frame sequence relationship of each frame of the first target image in a plurality of frames of the first target image;

[0034] Based on the frame sequence relationship of each frame of the first target image, the corresponding target reconstructed images are synthesized in sequence to obtain the corresponding target reconstructed video.

[0035] The present application also provides an image reconstruction device, including:

[0036] An image acquisition module, used to acquire a first target image and a second target image respectively, wherein the first target image has a first target object, the second target image has a second target object, and the first target object and the second target object have different type information;

[0037] A feature extraction module, used to extract features from the first target image and the second target image to obtain object feature information corresponding to the first target object and the second target object;

[0038] An image reconstruction module is used to perform image reconstruction based on the object feature information to obtain a target reconstructed image, wherein the target reconstructed image has the second target object, and in the target reconstructed image the second target object has the posture information of the first target object.

[0039] Based on the above, an embodiment of the present application further provides an electronic device, including:

[0040] Memory for storing computer programs;

[0041] The processor connected to the memory is used to execute the computer program stored in the memory to implement the above-mentioned image reconstruction method.

[0042] On the basis of the above, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is run, each step included in the above-mentioned image reconstruction method is executed.

[0043] The image reconstruction method, image reconstruction device, electronic device and storage medium provided by the present application can obtain the first target image and the second target image corresponding to the first target object and the second target object having different types of information respectively, so that the target reconstructed image can be reconstructed based on the object feature information of the first target image and the second target image. Among them, since the reconstruction basis of the obtained target reconstructed image is the object feature information corresponding to the first target object and the second target object, the target reconstructed image can include the features of the first target object and the second target object at the same time, that is, it includes the features of objects of different types, so that compared with the conventional technical solution of fusing the features of objects of the same type in the process of image reconstruction, it can have a better reconstruction effect, thereby improving the problem of poor reconstruction effect in the process of reconstructing images in the prior art, realizing the linkage between images with different types of objects, and expanding the scope of application of image reconstruction.

[0044] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A structural block diagram of an electronic device provided in an embodiment of the present application.

[0046] Figure 2A flowchart of an image reconstruction method provided in an embodiment of the present application.

[0047] Figure 3 for Figure 2 Schematic diagram of the flow chart of the sub-steps included in step S120.

[0048] Figure 4 A schematic diagram of the network architecture of the target neural network provided in an embodiment of the present application.

[0049] Figure 5 A schematic diagram of other steps included in the image reconstruction method provided in an embodiment of the present application.

[0050] Figure 6 A block diagram of an image reconstruction device provided in an embodiment of the present application.

[0051] Icons: 10 - electronic device; 12 - memory; 14 - processor; 100 - image reconstruction device; 110 - image acquisition module; 120 - feature extraction module; 130 - image reconstruction module. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0053] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0054] like Figure 1 As shown, an embodiment of the present application provides an electronic device 10 , wherein the electronic device 10 may include a memory 12 , a processor 14 , and an image reconstruction device 100 .

[0055] Optionally, in an alternative example, the memory 12 and the processor 14 may be electrically connected directly or indirectly to achieve data transmission or interaction. For example, they may be electrically connected to each other via one or more communication buses or signal lines. The image reconstruction device 100 includes at least one software function module that can be stored in the memory 12 in the form of software or firmware. The processor 14 is used to execute an executable computer program stored in the memory 12, for example, the software function module and computer program included in the image reconstruction device 100, to implement the image reconstruction method provided in the embodiment of the present application.

[0056] Optionally, in an alternative example, the memory 12 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0057] Optionally, in an alternative example, the processor 14 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a system on chip (SoC), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0058] Understandably, Figure 1 The structure shown is for illustration only. The electronic device 10 may also include Figure 1 More or fewer components as shown, or with Figure 1 The different configurations shown, for example, may also include a communication unit for exchanging information with other devices (such as a database).

[0059] Optionally, in an alternative example, the electronic device 10 may be a server with data processing capability.

[0060] Combination Figure 2The present application also provides an image reconstruction method applicable to the electronic device 10. The steps defined in the process related to the image reconstruction method can be implemented by the electronic device 10. Figure 2 The specific process shown is explained in detail.

[0061] Step S110, acquiring a first target image and a second target image respectively.

[0062] In this embodiment, the electronic device 10 may first obtain a first target image and a second target image respectively, wherein the first target image has a first target object, the second target image has a second target object, and the first target object and the second target object have different type information.

[0063] Step S120: extract object features from the first target image and the second target image to obtain object feature information corresponding to the first target object and the second target object.

[0064] In this embodiment, after obtaining the first target image and the second target image based on step S110, the electronic device 10 can perform object feature extraction on the first target image and the second target image, so as to obtain object feature information corresponding to the first target object and the second target object.

[0065] Step S130: reconstructing an image based on the object feature information to obtain a target reconstructed image.

[0066] In this embodiment, after obtaining the object feature information based on step S130, the electronic device 10 may perform image reconstruction based on the object feature information to obtain a target reconstructed image.

[0067] Among them, the target reconstructed image has the second target object, and in the target reconstructed image the second target object has the posture information of the first target object, so that the target reconstructed image has the feature information of the first target object and the second target object at the same time, realizing the fusion of feature information of different target objects.

[0068] Based on the above method, by obtaining the first target image and the second target image corresponding to the first target object and the second target object having different types of information respectively, the target reconstructed image can be reconstructed based on the object feature information of the first target image and the second target image. Among them, since the reconstruction basis of the obtained target reconstructed image is the object feature information corresponding to the first target object and the second target object, the target reconstructed image can include the features of the first target object and the second target object at the same time, that is, it includes the features of objects of different types, so that compared with the conventional technical solution of fusing the features of objects of the same type in the process of image reconstruction, it can have a better reconstruction effect, thereby improving the problem of poor reconstruction effect in the process of reconstructing images in the prior art, realizing the linkage between images with different types of objects, and expanding the scope of application of image reconstruction.

[0069] In the first aspect, it should be noted that for step S110, the method of acquiring the first target image and the second target image is not limited and can be selected according to actual application requirements.

[0070] For example, in an alternative example, the first target image and the second target image that are pre-saved may be acquired from a database server that is communicatively connected to the electronic device 10 .

[0071] For another alternative example, the first target image and the second target image can be obtained from a user terminal device that is communicatively connected to the electronic device 10. For example, when the user corresponding to the user terminal device needs to reconstruct the image, the user can respectively capture the first target image and the second target image and send them to the electronic device 10.

[0072] It can be understood that, in the above example, the different category information between the first target object and the second target object may mean that the first target object and the second target object belong to different species, such as a human and a cat, a human and a dog, or a cat and a dog, etc.

[0073] For example, in an application example, the first target object may be a person, and the second target object may be a cat. Thus, the reconstructed target image may be the cat in the second target image, and the cat has the posture information of the person in the first target image.

[0074] It can be understood that, in the above example, the second target object having the posture information of the first target object in the target reconstructed image may refer to facial expression posture information and / or posture information of other parts of the body.

[0075] It can be understood that in the above example, if the posture information is expression posture information, after obtaining the first target image and the second target image, in order to improve the accuracy of the reconstructed image, the first target image and the second target image can also be cut so that the cut first target image and the second target image include as few images of other parts as possible on the basis of including the head image.

[0076] On the second aspect, it should be noted that for step S120, the manner of extracting object features from the first target image and the second target image is not limited and can be selected according to actual application requirements.

[0077] For example, in an alternative example, the pixel coordinates of the key points of the first target object and the second target object in the first target image and the second target image (such as 108 key points of a face, etc.) can be extracted to obtain corresponding object feature information.

[0078] For example, in another alternative example, in order to make the target reconstructed image reconstructed based on the extracted object feature information have a higher quality, Figure 3 Step 120 may include step S121, step S122 and step S123, the specific contents of which are described as follows.

[0079] Step S121, obtaining a first sample image and a second sample image.

[0080] In this embodiment, a first sample image and a second sample image may be obtained first. The objects in the first sample image have the same category information as the first target object, for example, both may be people. The objects in the second sample image have the same category information as the second target object, for example, both may be cats.

[0081] Step S122: obtaining an image reconstruction model through training based on the first sample image and the second sample image.

[0082] In this embodiment, after the first sample image and the second sample image are acquired based on step S121, an image reconstruction model may be obtained by training based on the first sample image and the second sample image. The image reconstruction model may be a neural network model.

[0083] Step S123: Process the first target image and the second target image based on the image reconstruction model to obtain object feature information corresponding to the first target object and the second target object.

[0084] In this embodiment, after the image reconstruction model is obtained based on step S122, the first target image and the second target image can be processed based on the image reconstruction model to obtain object feature information corresponding to the first target object and the second target object.

[0085] Optionally, in an alternative example, when executing step S121, multiple frames of the first sample image and the multiple frames of the second sample image can be acquired respectively, so that when training the image reconstruction model, there can be sufficient samples to ensure that the trained image reconstruction model can have a higher image reconstruction effect.

[0086] For example, in an alternative example, a video to be processed having the first target object may be obtained first, and then the video to be processed may be cut along the time axis based on a scene change detection algorithm to obtain multiple video clips, wherein each video clip needs to ensure that there is no scene switching. Then, after obtaining multiple video clips, each video clip may be processed based on a pre-configured preprocessing algorithm to obtain a corresponding target video clip.

[0087] For example, in an alternative example, the first target object may be a person, and the preprocessing algorithm may include the following steps:

[0088] The first step is to obtain a video frame in the video clip as a target video frame;

[0089] The second step is to cut the target video frame based on the head detection algorithm to obtain the corresponding head area video frame, and extract the facial features of the head area video frame based on the facial feature extraction algorithm to obtain the corresponding target facial feature information;

[0090] The third step is to determine whether the target facial feature information belongs to the first facial feature information obtained, and if the target facial feature information belongs to the first facial feature information obtained, save the target facial feature information and save the target detection frame of the human body region video frame cut out; if the target facial feature information does not belong to the first facial feature information obtained, determine whether the target facial feature information and the first facial feature information obtained belong to the same person (such as calculating the matching degree between the facial feature information, etc.);

[0091] Step 4: If the target facial feature information and the first acquired facial feature information belong to the same person, then save the target detection frame of the human body region video frame obtained by cutting; if the target facial feature information and the first acquired facial feature information do not belong to the same person, then determine whether the number of video frames currently used as target video frames is greater than a pre-configured threshold;

[0092] Step 5: If the number of video frames currently used as target video frames is less than a pre-configured threshold, the next video frame in the video clip may be obtained as a new target video frame, and the above-mentioned steps 2, 3 and 4 are performed again based on the new target video frame;

[0093] In the sixth step, if the number of video frames that are currently used as target video frames is greater than or equal to a pre-configured threshold, the union of all currently saved target detection frames can be determined, and based on the union, the video frames that are currently used as target video frames can be cut to obtain corresponding multiple frames of new head area video frames to form the corresponding target video segment.

[0094] Based on the above steps, the obtained target video segment can include only the head area, only one person, and the number of included video frames is greater than or equal to a pre-configured threshold.

[0095] For example, in an alternative example, multiple frames (large number) of second sample images can be directly acquired. For another alternative example, multiple frames (small number) of second sample images can be acquired, and then, for each frame of the second sample image, a corresponding multiple-frame simulated second sample image is obtained based on a data enhancement algorithm to enhance the second sample image.

[0096] For example, in an application example, the object in the second sample image may be a cat. Considering that the algorithm for detecting the head region of a cat is not accurate enough, at least one frame of a second sample image of the head region of the cat may be obtained, and then, for each frame of the second sample image, the second sample image may be subjected to a radial change by randomly selecting a transformation matrix parameter from a Gaussian distribution based on a pre-configured data enhancement algorithm to obtain multiple frames of second sample simulation images corresponding to the second sample image, wherein each frame of the second sample simulation image is used as a frame of the second sample image obtained for model training.

[0097] Optionally, in an alternative example, when executing step S122, considering that the type information of the objects in the first sample image and the second sample image is different, and it is necessary to make the type information of the objects in the target reconstructed image reconstructed based on the trained image reconstruction model the same as the type information of the objects in the second sample image, therefore, step S122 may include the following sub-steps:

[0098] Sub-step 1, constructing a target neural network model;

[0099] Sub-step 2, reconstructing the first sample image and the second sample image based on the target neural network model to obtain a first reconstructed image to be processed, and reconstructing the acquired third sample image and the first reconstructed image to be processed based on the target neural network model to obtain a second reconstructed image to be processed, wherein the object in the third sample image is the same as the object in the second sample image;

[0100] Sub-step 3, calculating the corresponding image reconstruction loss based on the third sample image and the second reconstructed image to be processed, and updating the network parameters of the target neural network model based on the image reconstruction loss to obtain an image reconstruction model.

[0101] It is understood that in an alternative example, in combination with Figure 4 The target neural network model constructed based on sub-step 1 in the above example may include a first detection network, a second detection network, a dense motion network, a feature encoding network and an image reconstruction network.

[0102] In detail, the first detection network and the second detection network may be networks of the same architecture, such as both being encoder-decoder networks, such as Keypoint Detector. The dense motion network may be a Dense motion network. The feature encoding network may be an Encoder. The image reconstruction network may be a Decoder.

[0103] It is understandable that in Figure 4 In the network architecture shown, the same network belongs to the same network, but the input information is different. For example, the second detection network can input the second sample image, the first to-be-processed reconstructed image and the third sample image respectively in one training.

[0104] It is understandable that in an alternative example, based on Figure 4 The target neural network model shown, sub-step 2 in the above example, may include the following sub-steps:

[0105] The first step is to perform key point detection on the first target image based on the first detection network included in the target neural network model to obtain first key point information corresponding to the first target image, and to perform key point detection on the second target image based on the second detection network included in the target neural network model to obtain second key point information corresponding to the second target image;

[0106] The second step is to process the first key point information and the second key point information based on the dense motion network included in the target neural network model to obtain corresponding dense optical flow information and occlusion map information, and to perform feature encoding on the second target image based on the feature encoding network included in the target neural network model to obtain corresponding feature vector information;

[0107] The third step is to perform a target vector processing operation based on the dense optical flow information, the occlusion map information and the feature vector information to obtain corresponding target feature vector information, and to perform image reconstruction on the target feature vector information based on the image reconstruction network included in the target neural network model to obtain a corresponding first reconstructed image to be processed;

[0108] In the fourth step, based on the second detection network, key point detection is performed on the acquired third sample image and the first reconstructed image to be processed respectively to obtain third key point information corresponding to the third sample image and fourth key point information corresponding to the first reconstructed image to be processed, and the third key point information and the fourth key point information are processed based on the dense motion network to obtain corresponding new dense optical flow information and new occlusion map information;

[0109] The fifth step is to perform feature encoding on the first to-be-processed reconstructed image based on the feature encoding network to obtain corresponding new feature vector information, and perform target vector processing operation based on the new dense optical flow information, the new occlusion map information and the new feature vector information to obtain corresponding new target feature vector information;

[0110] The sixth step is to reconstruct the new target feature vector information based on the image reconstruction network to obtain a corresponding second reconstructed image to be processed.

[0111] It can be understood that, in an alternative example, in the second step of the above example, before processing the first key point information and the second key point information based on the dense motion network, affine transformations can be performed on the first key point information and the second key point information respectively to obtain corresponding first order motion representation information, and then, the corresponding first order motion representation information is processed based on the dense motion network to obtain the dense optical flow information and the occlusion map information.

[0112] The dense optical flow information may represent a mapping relationship between each key point in the first sample image and the second sample image, and the occlusion map information may represent an area where the corresponding feature map should be inpainted.

[0113] It can be understood that, in an alternative example, in the third step in the above example, the target vector processing operation may include the following sub-steps:

[0114] First, a grid sampling operation is performed based on the feature vector information and the dense optical flow information, and then an element-wise vector multiplication is performed based on the result of the operation and the occlusion map information to obtain the target feature vector information.

[0115] It is understandable that, in an alternative example, a specific form of the image reconstruction loss calculated based on sub-step 3 in the above example is perceptual loss. The algorithm for calculating the image reconstruction loss may include passing the third sample image and the second reconstructed image to be processed through VGG19 (a convolutional neural network that may include 16 convolutional layers and 3 fully connected layers), taking the eigenvalues ​​of 5 layers and performing difference calculations, then calculating the average value of the difference, and finally summing them to obtain the image reconstruction loss.

[0116] It can be understood that, based on the above example, in an alternative example, in order to make the target reconstructed image reconstructed based on the image reconstruction model obtained through training have the object in the second sample image, therefore, during the training process, after obtaining the second reconstructed image to be processed, it is also possible to determine whether the object in the second reconstructed image to be processed belongs to the object in the second sample image through a pre-configured object discriminator (such as Discriminator), that is, calculate the probability information that the object in the second reconstructed image to be processed belongs to the object in the second sample image, and then, calculate the corresponding loss value based on a loss calculation algorithm (such as least square GAN), and update the network parameters of the target neural network model based on the loss value, so that the overlapping loss values ​​converge, thereby completing the training.

[0117] It can be understood that, based on the above example, in an alternative example, in order to improve the detection accuracy of the second detection network, step S122 may also include the following sub-steps:

[0118] Sub-step 4, after obtaining the third key point information corresponding to the third sample image, calculating the corresponding equivariance loss based on the third key point information;

[0119] Sub-step 5: updating the network parameters of the second detection network based on the equivariant loss.

[0120] In this embodiment, after the third sample image is subjected to key point detection by the second detection network based on the aforementioned steps to obtain the corresponding third key point information, the loss formed by the key point detection can be calculated based on the third key point information, such as the corresponding equivariance loss. Then, the network parameters of the second detection network can be updated based on the equivariance loss so that the final loss converges.

[0121] Optionally, in an alternative example, in order to extract object feature information including relatively sufficient features of the first target object and the second target object so as to achieve a better subsequent image reconstruction effect, step S123 may include the following sub-steps:

[0122] The first step is to perform key point detection on the first target image based on the first detection network included in the image reconstruction model to obtain first key point information corresponding to the first target image, and to perform key point detection on the second target image based on the second detection network included in the image reconstruction model to obtain second key point information corresponding to the second target image;

[0123] The second step is to process the first key point information and the second key point information based on the dense motion network included in the image reconstruction model to obtain corresponding dense optical flow information and occlusion map information, and to perform feature encoding on the second target image based on the feature encoding network included in the image reconstruction model to obtain corresponding feature vector information;

[0124] In the third step, a target vector processing operation is performed based on the dense optical flow information, the occlusion map information and the feature vector information to obtain corresponding target feature vector information, and the target feature vector information is used as the object feature information corresponding to the first target object and the second target object.

[0125] It can be understood that the specific content of each sub-step included in step S123 in the above example can refer to the explanation of each network in the training process of the image reconstruction model in the previous text.

[0126] Secondly, it should be noted that for step S130, the specific method of performing image reconstruction based on the object feature information to obtain the target reconstructed image is not limited. For example, different image reconstruction methods may be used based on different representation forms of the object feature information. For example, when the object feature information is target feature vector information extracted based on the image reconstruction model, the target feature vector information may be reconstructed based on the image reconstruction network included in the image reconstruction model to obtain a corresponding target reconstructed image.

[0127] Based on the above example, in an alternative example, the first target image may be multiple frames, and each frame of the first target image is reconstructed with the second target image to obtain a corresponding frame of target reconstructed image. For example, for each frame of the first target image, based on the first target image and the second target image, by executing the above steps S120 and S130, a frame of target reconstructed image corresponding to the first target image can be obtained.

[0128] Based on this, for multiple frames of the first target image, corresponding multiple frames of the target reconstructed image can be obtained. On this basis, combined with Figure 5 , the image reconstruction method may further include the following steps:

[0129] Step S140: determining a frame sequence relationship of each frame of the first target image in a plurality of frames of the first target image.

[0130] In this embodiment, the electronic device 10 can also determine the frame sequence relationship of each frame of the first target image in the acquired multiple frames of the first target image. For example, multiple frames of the first target image can constitute a video clip, and the frame sequence relationship is used to represent the frame number of each first target image in the video clip.

[0131] Step S150, based on the frame sequence relationship of each frame of the first target image, the corresponding target reconstructed images are synthesized in sequence to obtain a corresponding target reconstructed video.

[0132] In this embodiment, after obtaining the frame sequence relationship based on step S140, the electronic device 10 can also synthesize the corresponding target reconstructed images in sequence based on the frame sequence relationship of each frame of the first target image, such as arranging and combining them according to the corresponding frame numbers to obtain the corresponding target reconstructed video.

[0133] Based on this, the above-mentioned steps S140 and S150 can transfer the dynamic posture information of an object in a video clip to other objects to form a corresponding target reconstructed video, such as transferring the expression changes of a person in a video clip to a target reconstructed video of a cat, thereby realizing the driving of the cat's expression based on the human expression video.

[0134] Combination Figure 6 The embodiment of the present application further provides an image reconstruction device 100 applicable to the electronic device 10. The image reconstruction device 100 may include an image acquisition module 110, a feature extraction module 120 and an image reconstruction module 130.

[0135] The image acquisition module 110 is used to acquire a first target image and a second target image respectively, wherein the first target image has a first target object, the second target image has a second target object, and the first target object and the second target object have different type information. In this embodiment, the image acquisition module 110 can be used to perform Figure 2 As shown in step S110 , the relevant contents of the image acquisition module 110 may refer to the above description of step S110 .

[0136] The feature extraction module 120 is used to extract features from the first target image and the second target image to obtain object feature information corresponding to the first target object and the second target object. Figure 2 As shown in step S120, the relevant contents of the feature extraction module 120 can refer to the above description of step S120.

[0137] The image reconstruction module 130 is used to perform image reconstruction based on the object feature information to obtain a target reconstructed image, wherein the target reconstructed image has the second target object, and in the target reconstructed image the second target object has the posture information of the first target object. In this embodiment, the image reconstruction module 130 can be used to perform Figure 2 As shown in step S130, the relevant contents of the image reconstruction module 130 can refer to the above description of step S130.

[0138] In an embodiment of the present application, corresponding to the above-mentioned image reconstruction method applied to the electronic device 10, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is run, the various steps of the image reconstruction method are executed, such as the above-mentioned step S110, step S120 and step S130.

[0139] The steps executed when the aforementioned computer program is running will not be described one by one here, and reference may be made to the explanation of the image reconstruction method in the previous text.

[0140] In summary, the image reconstruction method, image reconstruction device, electronic device and storage medium provided by the present application can reconstruct a target reconstructed image based on the object feature information of the first target image and the second target image by acquiring the first target image and the second target image corresponding to the first target object and the second target object having different types of information respectively. Among them, since the reconstruction basis of the obtained target reconstructed image is the object feature information corresponding to the first target object and the second target object, the target reconstructed image can include the features of the first target object and the second target object at the same time, that is, it includes the features of objects of different types, so that compared with the conventional technical solution of fusing the features of objects of the same type in the process of image reconstruction, it can have a better reconstruction effect, thereby improving the problem of poor reconstruction effect in the process of reconstructing images in the prior art, realizing the linkage between images with different types of objects, and expanding the scope of application of image reconstruction.

[0141] In several embodiments provided in the embodiments of the present application, it should be understood that the disclosed device and method can also be implemented in other ways. The device and method embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the device, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0143] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk. It should be noted that, in this article, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0144] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image reconstruction method, characterized in that: include: Acquire a first target image and a second target image respectively, wherein the first target image has a first target object, the second target image has a second target object, and the first target object and the second target object have different type information; Acquire a first sample image and a second sample image, wherein the object in the first sample image has the same category information as the first target object, and the object in the second sample image has the same category information as the second target object; construct a target neural network model; reconstruct the first sample image and the second sample image based on the target neural network model to obtain a first reconstructed image to be processed, and reconstruct the acquired third sample image and the first reconstructed image to be processed based on the target neural network model to obtain a second reconstructed image to be processed, wherein the object in the third sample image is the same as the object in the second sample image; calculate the corresponding image reconstruction loss based on the third sample image and the second reconstructed image to be processed, and update the network parameters of the target neural network model based on the image reconstruction loss to obtain an image reconstruction model; Processing the first target image and the second target image based on the image reconstruction model to obtain object feature information corresponding to the first target object and the second target object; An image is reconstructed based on the object feature information to obtain a target reconstructed image, wherein the target reconstructed image has the second target object, and in the target reconstructed image the second target object has the posture information of the first target object.

2. The image reconstruction method according to claim 1, characterized in that: The step of reconstructing the first sample image and the second sample image based on the target neural network model to obtain a first reconstructed image to be processed, and reconstructing the acquired third sample image and the first reconstructed image to be processed based on the target neural network model to obtain a second reconstructed image to be processed includes: Performing key point detection on the first target image based on a first detection network included in the target neural network model to obtain first key point information corresponding to the first target image, and performing key point detection on the second target image based on a second detection network included in the target neural network model to obtain second key point information corresponding to the second target image; Processing the first key point information and the second key point information based on the dense motion network included in the target neural network model to obtain corresponding dense optical flow information and occlusion map information, and performing feature encoding on the second target image based on the feature encoding network included in the target neural network model to obtain corresponding feature vector information; Performing a target vector processing operation based on the dense optical flow information, the occlusion map information, and the feature vector information to obtain corresponding target feature vector information, and performing image reconstruction on the target feature vector information based on an image reconstruction network included in the target neural network model to obtain a corresponding first reconstructed image to be processed; Based on the second detection network, key point detection is performed on the acquired third sample image and the first to-be-processed reconstructed image respectively to obtain third key point information corresponding to the third sample image and fourth key point information corresponding to the first to-be-processed reconstructed image, and the third key point information and the fourth key point information are processed based on the dense motion network to obtain corresponding new dense optical flow information and new occlusion map information; Based on the feature encoding network, feature encoding is performed on the first to-be-processed reconstructed image to obtain corresponding new feature vector information, and a target vector processing operation is performed based on the new dense optical flow information, the new occlusion map information and the new feature vector information to obtain corresponding new target feature vector information; Image reconstruction is performed on the new target feature vector information based on the image reconstruction network to obtain a corresponding second reconstructed image to be processed.

3. The image reconstruction method according to claim 2, characterized in that: The step of obtaining an image reconstruction model through training based on the first sample image and the second sample image comprises: After obtaining the third key point information corresponding to the third sample image, calculating the corresponding equivariance loss based on the third key point information; The network parameters of the second detection network are updated based on the equivariant loss.

4. The image reconstruction method according to claim 1, characterized in that: The step of processing the first target image and the second target image based on the image reconstruction model to obtain object feature information corresponding to the first target object and the second target object includes: Performing key point detection on the first target image based on a first detection network included in the image reconstruction model to obtain first key point information corresponding to the first target image, and performing key point detection on the second target image based on a second detection network included in the image reconstruction model to obtain second key point information corresponding to the second target image; Processing the first key point information and the second key point information based on a dense motion network included in the image reconstruction model to obtain corresponding dense optical flow information and occlusion map information, and performing feature encoding on the second target image based on a feature encoding network included in the image reconstruction model to obtain corresponding feature vector information; A target vector processing operation is performed based on the dense optical flow information, the occlusion map information and the feature vector information to obtain corresponding target feature vector information, and the target feature vector information is used as object feature information corresponding to the first target object and the second target object.

5. The image reconstruction method according to any one of claims 1 to 4, characterized in that: The first target image is a plurality of frames, and each frame of the first target image is reconstructed with the second target image to obtain a corresponding frame of target reconstructed image. The image reconstruction method further includes: Determining a frame sequence relationship of each frame of the first target image in a plurality of frames of the first target image; Based on the frame sequence relationship of each frame of the first target image, the corresponding target reconstructed images are synthesized in sequence to obtain the corresponding target reconstructed video.

6. An image reconstruction device, characterized in that: include: An image acquisition module, used to acquire a first target image and a second target image respectively, wherein the first target image has a first target object, the second target image has a second target object, and the first target object and the second target object have different type information; A feature extraction module, used to obtain a first sample image and a second sample image, wherein the object in the first sample image has the same category information as the first target object, and the object in the second sample image has the same category information as the second target object; construct a target neural network model; reconstruct the first sample image and the second sample image based on the target neural network model to obtain a first reconstructed image to be processed, and reconstruct the acquired third sample image and the first reconstructed image to be processed based on the target neural network model to obtain a second reconstructed image to be processed, wherein the object in the third sample image is the same as the object in the second sample image; calculate the corresponding image reconstruction loss based on the third sample image and the second reconstructed image to be processed, and update the network parameters of the target neural network model based on the image reconstruction loss to obtain an image reconstruction model; process the first target image and the second target image based on the image reconstruction model to obtain object feature information corresponding to the first target object and the second target object; An image reconstruction module is used to perform image reconstruction based on the object feature information to obtain a target reconstructed image, wherein the target reconstructed image has the second target object, and in the target reconstructed image the second target object has the posture information of the first target object.

7. An electronic device, characterized in that: include: Memory for storing computer programs; A processor connected to the memory is used to execute the computer program stored in the memory to implement the image reconstruction method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is run, each step included in the image reconstruction method according to any one of claims 1 to 5 is executed.

Images