Training of oil painting reconstruction network and image reconstruction and video reconstruction method and device thereof

Abstract

The application discloses a kind of oil painting reconstruction network training and its image reconstruction, video reconstruction method and equipment, the method comprises the following steps: obtaining the first sample image data with the content of real scene;Obtain the second sample image data with the content of oil painting;If the second sample image data converges to the oil painting style of a certain age, then add new second sample image data to balance the oil painting style of different ages;With the first sample image data as the source of oil painting content, the second sample image data is trained to generate an adversarial network, and an oil painting reconstruction network is obtained.The samples of the oil painting style of different ages are balanced, which can improve the performance of the oil painting reconstruction network, avoid the problems caused by the oil painting style of a single age, and the oil painting reconstruction network belongs to the generative adversarial network, which can reconstruct the image data to the oil painting style.The oil painting style reconstruction belongs to post-processing, which greatly improves the efficiency of producing video data with oil painting style.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a training method and device of a painting reconstruction network, and an image reconstruction and video reconstruction method and device. BACKGROUND

[0002] In short video, advertisement and other scenarios, users will make various types of video data, and after recording original video data, post-processing is usually performed on the video data to improve the quality of the video data.

[0003] Due to certain business needs, part of the post-processing is to convert the style of the video data to a painting style, and the commonly used post-processing is to add a filter to the video data to convert the entire video data to other styles, such as retro, film, sunset, etc.

[0004] However, the filter usually adjusts the color value of the pixel point and adds other decorative elements, and the effect is relatively single, and it is difficult to achieve the style of some specific paintings by using multiple filters to superimpose. If the video data is designed according to the style of a specific painting, it will greatly increase the threshold for producing video data, resulting in a significant extension of the time-consuming for producing video data and low efficiency for producing video data. SUMMARY

[0005] The present application provides a training method and device of a painting reconstruction network, and an image reconstruction and video reconstruction method and device to solve the problem of efficiently implementing the style of a painting in video data.

[0006] According to an aspect of the present application, a training method of a painting reconstruction network is provided, comprising:

[0007] obtaining first sample image data with content of a real scene;

[0008] obtaining second sample image data with content of a painting;

[0009] if the second sample image data converges to a painting style of a certain era, adding new second sample image data to balance the painting styles of different eras;

[0010] training a generative adversarial network with the first sample image data as the source of painting content and the second sample image data as the source of painting style to obtain a painting reconstruction network.

[0011] According to another aspect of the present application, an image reconstruction method is provided, comprising:

[0012] loading a painting reconstruction network trained according to the method of any embodiment of the present application;

[0013] detecting a target element that is not suitable for a painting style in volume in original image data to be reconstructed;

[0014] reconstructing the original image data into candidate image data containing a painting style by inputting the original image data into the painting reconstruction network;

[0015] pasting the target element back into the candidate image data to obtain target image data.

[0016] According to another aspect of the present application, a video reconstruction method is provided, comprising:

[0017] loading a painting reconstruction network trained according to the method of any embodiment of the present application;

[0018] obtaining original video data with content of introducing a game, the original video data having multiple frames of original image data, at least part of the original image data having a target element that is not suitable for a painting style in volume;

[0019] reconstructing the original image data into candidate image data containing a painting style by inputting the original image data into the painting reconstruction network;

[0020] pasting the target element back into the candidate image data to obtain target image data;

[0021] replacing the original image data with the target image data in the original video data to obtain target video data.

[0022] According to another aspect of the present application, an electronic device is provided, comprising:

[0023] at least one processor; and

[0024] a memory in communication connection with the at least one processor; wherein,

[0025] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the training method of the painting reconstruction network or the image reconstruction method or the video reconstruction method according to any embodiment of the present application.

[0026] According to another aspect of the present application, a computer readable storage medium is provided, the computer readable storage medium storing a computer program, the computer program being used to enable a processor to implement the training method of the painting reconstruction network or the image reconstruction method or the video reconstruction method according to any embodiment of the present application when executed by the processor.

[0027] In the embodiment, the first sample image data is obtained from a real scene, and the second sample image data is obtained from a painting; if the second sample image data converges to a painting style of a certain era, new second sample image data is added to balance painting styles of different eras; a generative adversarial network is trained with the first sample image data as the source of painting content and the second sample image data as the source of painting style, to obtain a painting reconstruction network. Balancing the sample of painting styles of different eras can improve the performance of the painting reconstruction network, avoid the problems caused by a single era of painting style, and the painting reconstruction network belongs to the generative adversarial network, which can reconstruct image data into a painting style. The reconstruction of the painting style belongs to post-processing, which can maintain the threshold of producing video data and the time-consuming of producing video data, and greatly improves the efficiency of producing video data of the painting style.

[0028] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the application, nor is it intended to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0030] Figure 1 is a flowchart of a training method of a painting reconstruction network according to the first embodiment of the application;

[0031] Figure 2 is a structural schematic diagram of a classifier according to the first embodiment of the application;

[0032] Figure 3 is a flowchart of a training method of a painting reconstruction network according to the second embodiment of the application;

[0033] Figure 4 is a flowchart of an image reconstruction method according to the third embodiment of the application;

[0034] Figures 5A to 5C is an example diagram of image reconstruction according to the third embodiment of the application;

[0035] Figure 6 is a flowchart of a video reconstruction method according to the fourth embodiment of the application;

[0036] Figure 7is a structural schematic diagram of an oil painting reconstruction network training device provided according to an embodiment five of the present application;

[0037] Figure 8 is a structural schematic diagram of an image reconstruction device provided according to an embodiment six of the present application;

[0038] Figure 9 is a structural schematic diagram of a video reconstruction device provided according to an embodiment seven of the present application;

[0039] Figure 10 is a structural schematic diagram of an electronic device provided according to an embodiment eight of the present application. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts should fall within the scope of the present application.

[0041] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product or device.

[0042] Embodiment one

[0043] Figure 1 A flowchart of an oil painting reconstruction network training method provided for the embodiment one of the present application, the embodiment can be applicable to the case of training an oil painting reconstruction network realizing an oil painting style, the method can be executed by an oil painting reconstruction network training device, the oil painting reconstruction network training device can be realized in the form of hardware and / or software, and the oil painting reconstruction network training device can be configured in an electronic device. As shown in the figure, the method comprises: Figure 1

[0044] Step 101, obtaining first sample image data with content being a real scene. ​

[0045] In the embodiment, the image data of the real scene can be collected from the image bed, a public data set, and the like, and is denoted as first sample image data.

[0046] The real scene can include a real natural environment, a real building, a real person, and the like, and the embodiment is not limited in this regard.

[0047] In step 102, second sample image data of oil paintings is obtained.

[0048] An oil painting is a kind of painting made of fast-drying plant oil (such as linseed oil, walnut oil, and the like) mixed with pigments on a canvas, linen, paperboard, or wooden board. The diluent used for painting is volatile turpentine and dry linseed oil, and the like. The pigments attached to the painting have strong hardness and can maintain luster for a long time after the painting is dry. The pigments have sufficient hiding power and transparency to fully represent the object, and the painting has rich colors and strong three-dimensional texture.

[0049] In the embodiment, the image data of the oil painting can be collected from the image bed, a public data set, and the like, and is denoted as second sample image data, that is, the second sample image data presents the oil painting in the form of a digital image.

[0050] In one way, one or more oil painting data sets can be obtained, and the oil painting data set stores a plurality of frames of image data of oil paintings, for example, monet2photo, vangogh2photo, ukiyoe2photo, cezanne2photo, and the like.

[0051] One or more painters, especially painters familiar to the general public, are determined.

[0052] The image data of the oil painting painted by the painter is filtered out from the oil painting data set as second image data by taking the name of the painter as a keyword.

[0053] In step 103, if the second sample image data converges to the oil painting style of a certain era, new second sample image data is added to balance the oil painting styles of different eras.

[0054] In the embodiment, the distribution of the style of the second sample image data in each era can be analyzed. Due to the data set and the like, the second sample image data can converge to the oil painting style of a certain era, that is, a large amount of second sample image data belongs to the oil painting style of a certain era, which can cause some features to be aggregated, cause the oil painting reconstruction network to have some obvious biases, and the biases can have a certain gap with the aesthetics of modern users, and the biases can have a large deviation and easily appear distorted.

[0055] For example, a certain period of oil painting will avoid depicting smiles for some reason, if these oil paintings are used to train the oil painting reconstruction network, it will make the oil painting reconstruction network unable to present a smile in the reconstructed image data, even if the portrait in the image data as the content source has a smile, it is often unable to reproduce the smile after reconstruction.

[0056] In addition to smiles, features such as hairstyles, skin colors, and lines may deviate greatly from the original image data.

[0057] Therefore, if the second sample image data converges to the style of oil paintings of a certain period, new second sample image data can be added in other ways to balance the styles of oil paintings of different periods in the sample, that is, to make the styles of oil paintings of different periods balanced.

[0058] In an embodiment of the present application, step 103 can include the following steps:

[0059] Step 1031, query the painter who drew the oil painting represented by the second sample image data.

[0060] When collecting the second sample image data, the painter who drew the oil painting represented by the second sample image data can be recorded, at this time, the information of the painter, such as name, ID, etc., can be queried.

[0061] Step 1032, if the time when the painter drew the oil painting converges to a certain period, perform style processing on the first sample image data to convert the style of the first sample image data to the style of the oil painting.

[0062] For different painters, the period when they drew the oil painting can be queried, in order to save resources, the period when the painter lived can be equivalent to the period when he drew all the oil paintings.

[0063] If the time when these painters drew the oil painting converges to a certain period, that is, the time when these painters drew the oil painting is mainly concentrated in a certain period, one or more operators can be used to perform style processing on the first sample image data, that is, the pixel points of the first sample image data are processed according to certain logic, so that the style of the first sample image data is converted to the style of the oil painting.

[0064] In an example, the style of the first sample image data can be converted from real to oil painting as new second sample image data, which can form paired data and improve the quality of the sample, thereby improving the performance of the oil painting reconstruction network.

[0065] In this example, the first sample image data can be grayscale, and the first sample image data can be converted to grayscale image data.

[0066] The original gray value of the gray image data is normalized to a specified range [0, N] to obtain a candidate gray value.

[0067] A window is added to the gray image data (equivalent to adding a window to the first sample image data), and the window has a fixed size, such as 4x4, 6x6, etc. The window is moved in a preset direction on the gray image data (first sample image data) by a preset step size each time.

[0068] The number of candidate gray values in the window is counted each time the window is moved, and the candidate gray value with the largest number is selected as the target gray value.

[0069] The color value of the target pixel point in the first sample image data is averaged, wherein the target pixel point is the pixel point located at the position of the target gray value in the first sample image data, and the average value is used to replace the color value of the target pixel point.

[0070] When the gray image data (first sample image data) is traversed, the style of the first sample image data can be converted to an oil painting style.

[0071] Step 1033, adding the first sample image data with the oil painting style as new second sample image data.

[0072] The first sample image data with the oil painting style can be added to the set of second sample image data and labeled as second sample image data.

[0073] In another embodiment of the present application, step 103 can include the following steps:

[0074] Step 1034, collecting unlabeled image data as unknown image data.

[0075] In this embodiment, a large amount of unlabeled image data can be collected through unsorted channels, which is recorded as unknown image data. Therefore, the unknown image data can contain image data of real scenes, image data of oil paintings, image data of sketches, etc.

[0076] The unknown image data is identified to select unknown image data with oil painting content. Of course, in order to improve efficiency, some oil painting-related channels such as user-sharing oil painting websites and art websites can be downloaded to improve the probability of selecting unknown image data with oil painting content.

[0077] Step 1035, loading a plurality of classifiers for binary classification.

[0078] In a specific implementation, the image data to be reconstructed in the oil painting style contains a large amount of information, and the oil painting style is generally reconstructed to improve the degree of personalization and avoid excessive homogenization. Therefore, the oil painting reconstruction network can be trained by taking a modern realistic oil painting as a sample. On the one hand, the modern oil painting is more in line with the aesthetic of users, and on the other hand, the realistic oil painting has a low degree of abstraction, which is beneficial to the preservation of information of the image data.

[0079] In a specific implementation, a plurality of binary classification classifiers can be trained by taking a modern realistic oil painting as a target. Each classifier is used to implement part of the identification of the modern realistic oil painting, and all the classifiers can collectively implement the identification of the modern realistic oil painting.

[0080] The so-called binary classification can refer to a classification task with two categories. The input is represented by a feature vector x, and the output is represented by y=0 or 1.

[0081] Step 1036, sequentially calling the plurality of classifiers to label the unknown image data with a binary classification label.

[0082] In the embodiment, to achieve the goal of identifying a modern realistic oil painting, a logical order can be set for the classifiers, and the plurality of classifiers can be sequentially called to label the unknown image data with a binary classification label according to the logical order.

[0083] To reduce the calculation amount of the classifier at the back of the order, if the label of the current classifier does not conform to the modern realistic oil painting, the unknown image data can be filtered out and not entered into the next classifier. Otherwise, if the label of the current classifier conforms to the modern realistic oil painting, the unknown image data is entered into the next classifier for further processing.

[0084] In a specific implementation, the classifiers include a first classifier, a second classifier, and a third classifier. The first classifier is used to identify whether the image data is an oil painting, the second classifier is used to identify whether the image data is realistic, and the third classifier is used to identify whether the image data is modern.

[0085] Then, the first classifier can be called to label the unknown image data with a label indicating whether the content is an oil painting.

[0086] If the label of the unknown image data includes a non-oil painting, the unknown image data is filtered out.

[0087] If the label of the unknown image data includes an oil painting, the second classifier is then called to label the unknown image data with a label indicating whether the content is realistic.

[0088] If the label of the unknown image data includes a non-realistic oil painting, the unknown image data is filtered out.

[0089] If the label of the unknown image data includes a realistic oil painting, the third classifier is finally called to label the content of the unknown image data as modern or not.

[0090] Further, the first classifier, the second classifier and the third classifier can be independent classifiers, especially machine learning models such as SMV (Support Vector Machine) and the like, but the first classifier, the second classifier and the third classifier are correlated, and the machine learning model processes the unknown image data independently each time, so that the first classifier, the second classifier and the third classifier have overlapping workload, and the efficiency is reduced.

[0091] To this end, the first classifier, the second classifier and the third classifier can be constructed using deep learning in the embodiment, wherein, as shown in the figure, Figure 2 the first classifier has a first feature extraction layer and a first mapping layer, the second classifier has a second feature extraction layer and a second mapping layer, and the third classifier has a third feature extraction layer and a third mapping layer, the first feature extraction layer, the second feature extraction layer and the third feature extraction layer are some convolutional layers, residual networks, skip links and the like, and the first mapping layer, the second mapping layer and the third mapping layer can be fully connected layers, sofmax layers and the like.

[0092] When the first classifier labels the label of the unknown image data, the unknown image data is input into the first feature extraction layer to extract first image features; the first image features are input into the first mapping layer to map the label of whether the content of the unknown image data is an oil painting.

[0093] When the second classifier labels the label of the unknown image data, the first image features extracted by the first classifier are reused, and specifically, the first image features are input into the second feature extraction layer to extract second image features; the second image features are input into the second mapping layer to map the label of whether the content of the unknown image data is realistic.

[0094] When the third classifier labels the label of the unknown image data, the second image features extracted by the second classifier are reused, and specifically, the second image features are input into the third feature extraction layer to extract third image features; the third image features are input into the third mapping layer to map the label of whether the content of the unknown image data is modern.

[0095] In the embodiment, the second classifier reuses the first image features extracted by the first classifier, and the third classifier reuses the second image features extracted by the second classifier. By reusing the features in sequence, the operation amount of the features extracted subsequently can be reduced, so that the overall efficiency is improved.

[0096] In step 1037, the unknown image data labeled as modern realistic oil painting is added as new second sample image data.

[0097] For the unknown image data with the content of modern realistic oil painting, the image data can be added to the set of second sample image data and labeled as second sample image data.

[0098] The embodiment converges the range focused by each classifier to achieve the goal of identifying modern realistic oil painting. Each classifier plays a role in layer-by-layer screening, which can not only improve the identification rate but also improve the identification accuracy.

[0099] Of course, the above-mentioned manner of adding new second sample image data is only an example. In the implementation of the embodiment, other manners of adding new second sample image data can be set according to actual conditions, and the embodiment does not limit this. In addition, in addition to the above-mentioned manner of adding new second sample image data, other manners of adding new second sample image data can also be used by those skilled in the art according to actual needs, and the embodiment does not limit this.

[0100] In step 104, the generative adversarial network is trained by taking the first sample image data as the source of the oil painting content and the second sample image data as the source of the oil painting style, to obtain an oil painting reconstruction network.

[0101] In the embodiment, the generative adversarial network (GAN) can be constructed in advance.

[0102] Generally, the generative adversarial network includes a generator and a discriminator. The generator is responsible for generating content according to a random vector. In the embodiment, the content is image data, especially image data with an oil painting style. The discriminator is responsible for determining whether the received content is real. The discriminator usually gives a probability representing the authenticity of the content.

[0103] The generator and the discriminator can use different structures. For the function of processing image data, these structures are not limited to artificially designed neural networks, such as convolutional layers, fully connected layers, etc. They can also be neural networks optimized by model quantization methods, neural networks searched for oil painting styles by NAS (Neural Architecture Search), etc. The embodiment does not limit this.

[0104] The generative adversarial network can be divided into the following types according to different structures of the generator and the discriminator:

[0105] DCGAN (deep convolutional generative adversarial network), CGAN (conditional generative adversarial network), CycleGAN (cycle generative adversarial network), CoGAN (coupled generative adversarial network), ProGAN (progressive growth of generative adversarial network), WGAN (Wasserstein generative adversarial network), SAGAN (self-attention generative adversarial network), BigGAN (big generative adversarial network), StyleGAN (style-based generative adversarial network).

[0106] The generator and the discriminator are in confrontation, and the confrontation can refer to the process of alternating training of the generative adversarial network. Taking the generation of image data with an oil painting style as an example, the generator generates some false image data and true image data, which are given to the discriminator to distinguish between the two, and the discriminator learns to distinguish between the two, and gives a high score to the true image data (i.e., image data with an oil painting style) and a low score to the false image data (i.e., image data without an oil painting style). When the discriminator can skillfully judge the existing image data, the generator aims to obtain a high score from the discriminator and continuously generates better false image data until it can deceive the discriminator. The process is repeated until the prediction probability of the discriminator for any image data is close to 0.5, that is, the discriminator cannot distinguish between true and false image data, and the training is stopped.

[0107] In this embodiment, the first sample image data with a real scene content and the second sample image data with an oil painting content are used as samples for training the generative adversarial network. The first sample image data is the source of the oil painting content, and the second sample image data is the source of the oil painting style. The generative adversarial network is trained in this way, and the trained generative adversarial network is recorded as an oil painting reconstruction network. That is, the oil painting reconstruction network is used to reconstruct image data without an oil painting style into image data with an oil painting style.

[0108] Further, the samples for training the generative adversarial network can be paired data, which can improve the performance of the generative adversarial network, but this requires collecting image data of real scenes corresponding to the second sample image data. However, in fact, part of the second sample image data does not have corresponding image data of real scenes. Therefore, the generative adversarial network in this embodiment supports training with unpaired data, such as CycleGAN, StyleGAN, etc.

[0109] Taking Learning to Cartoonize Using White-box Cartoon Representations as an example, the network includes three modules, which can divide the original image and style image into three representations:

[0110] 1、surface (surface) representation

[0111] The surface representation is extracted to represent the smooth surface of the image data. Given the image data, the weighted low-frequency component can be extracted, in which the color component and the surface texture are retained, and the edges, textures and details are ignored, which can be used to realize a flexible and learnable feature representation of the smooth surface.

[0112] 2、structure (structure) representation

[0113] The structure representation can effectively capture the global structure information and sparse color blocks in the celluloid cartoon style. The segmentation area is extracted from the input image data, and an adaptive coloring algorithm is applied to each segmentation area to generate a structure representation. The structure representation can imitate the celluloid cartoon style, which is characterized by clear boundaries and sparse color blocks.

[0114] 3、texture (texture) representation

[0115] The texture representation includes the drawn details and edges. The input image data is converted into a single-channel intensity map, in which the color and brightness are removed and the relative pixel intensity is retained. The texture representation can guide the network to learn high-frequency texture details independently, excluding color and brightness patterns.

[0116] The weight of the surface representation, the structure representation and the texture representation is balanced to control the style of the output image data.

[0117] Further, for the characteristics of oil painting, the structure representation and its loss value can be deleted in the embodiment to improve the performance of the oil painting reconstruction network.

[0118] In the embodiment, the first sample image data is obtained from a real scene, and the second sample image data is obtained from a painting. If the second sample image data converges to a painting style of a certain era, new second sample image data is added to balance the painting styles of different eras. The generative adversarial network is trained with the first sample image data as the source of the painting content and the second sample image data as the source of the painting style, and the painting reconstruction network is obtained. The sample of the balanced painting styles of different eras can improve the performance of the painting reconstruction network, avoid the problems caused by a single era of painting style, and the painting reconstruction network belongs to the generative adversarial network, which can reconstruct the image data to the painting style. The reconstruction of the painting style belongs to the post-processing, which can maintain the threshold of producing video data and the time-consuming of producing video data, and greatly improves the efficiency of producing video data of the painting style.

[0119] Embodiment two

[0120] Figure 3 A flowchart of a training method of a painting reconstruction network is provided for the second embodiment of the application. The embodiment adds the operation of screening the first sample image data with bright colors on the basis of the above-mentioned embodiment. As shown in Figure 3 , the method comprises:

[0121] Step 301, obtaining first sample image data from a real scene.

[0122] Step 302, obtaining second sample image data from a painting.

[0123] Step 303, if the second sample image data converges to a painting style of a certain era, new second sample image data is added to balance the painting styles of different eras.

[0124] Step 304, if the painting expressed by the second sample image data has authenticity and typicality, the second sample image data is marked as seed image data.

[0125] For the second sample image data, its content (i.e. painting) can be checked for authenticity and typicality. Authenticity can mean that the painting is drawn by a painter and is not obtained by stylization processing. Typicality can mean that the painting can be a representative painting style among many painting styles. These representative paintings are generally drawn by famous painters, and the painter (represented by name, ID, etc.) can be pre-set for screening. That is, if some paintings are drawn by a pre-set painter, the second sample data with the painting content can be considered to have typicality.

[0126] If the painting expressed by the second sample image data has authenticity and typicality, the second sample image data can be marked as seed image data.

[0127] Step 305, respectively calculate the vividness of the first sample image data and the seed image data in color.

[0128] In the embodiment, a standard representing the vividness of image data in color (also referred to as color) can be constructed in advance, so that the vividness of the first sample image data in color and the vividness of the seed image data in color are respectively calculated according to the same standard.

[0129] For example, the first sample image data and the seed image data are respectively divided into red vectors R, green vectors G and blue vectors B in color. The red vectors R, the green vectors G and the blue vectors B are all vectors, not scalars.

[0130] The absolute value of the difference between the red vectors R and the green vectors G is taken to obtain a red-green vector rg, that is, rg = |R-G|.

[0131] The absolute value of the difference between the mixed vector and the blue vectors B is taken to obtain a yellow-blue vector yb, wherein the mixed vector is a specified first proportion α (such as 0.5) of the sum of the red vectors R and the green vectors G, that is, yb = |α*(R+G)-B|

[0132] The first standard deviation σ rg of the red-green vector is calculated. rg .

[0133] The second standard deviation σ yb of the yellow-blue vector is calculated. yb .

[0134] The square root of the sum of the square of the first standard deviation σ rg and the square of the second standard deviation σ yb is taken to obtain the third standard deviation σ rgyb , that is,

[0135] The square root of the sum of the square of the first average μ rg and the square of the second average μ yb is taken to obtain the third average μ rgyb ; that is,

[0136] The third average is multiplied by a specified second proportion β (such as 0.3) and added to the third standard deviation to obtain the vividness C, that is, C = σ rgyb + β * μ rgyb .

[0137] Step 306, construct a threshold based on the vividness of the seed image data in color.

[0138] In the embodiment, the brightness of the seed image data in color can be taken as a reference, and the threshold value can be constructed by calculating the evaluation value, the median value, the maximum value, and the like.

[0139] In step 307, if the brightness of the first sample image data in color is less than the threshold value, the first sample image data is filtered out.

[0140] The brightness of the first sample image data in color is compared with the threshold value. If the brightness of the first sample image data in color is greater than or equal to the threshold value, it indicates that the first sample image data is relatively bright in color, and the first sample image data can be retained. If the brightness of the first sample image data in color is less than the threshold value, it indicates that the first sample image data is relatively dark in color, and the first sample image data is filtered out.

[0141] In step 308, the generative adversarial network is trained by taking the first sample image data as the source of the oil painting content and the second sample image data as the source of the oil painting style, and an oil painting reconstruction network is obtained.

[0142] The first sample image data with relatively bright color is more equivalent to the second sample image data, and a sample with higher quality can be formed. The generative adversarial network is trained by taking the first sample image data with relatively bright color as the source of the oil painting content and the second sample image data as the source of the oil painting style, and the performance of the oil painting reconstruction network can be further improved.

[0143] Embodiment Three

[0144] Figure 4 A flowchart of an image reconstruction method provided for the third embodiment of the present application. The present embodiment can be applied to the case of reconstructing image data to an oil painting style based on an oil painting reconstruction network. The method can be performed by an image reconstruction device, which can be realized in the form of hardware and / or software, and can be configured in an electronic device. As shown in the figure, the method comprises: Figure 4

[0145] In step 401, an oil painting reconstruction network is loaded.

[0146] In a specific implementation, the oil painting reconstruction network can be trained in advance according to the method described in the first embodiment of the present application, wherein the oil painting reconstruction network is used to reconstruct image data of a non-oil painting style to image data of an oil painting style.

[0147] When the oil painting reconstruction network is applied, the oil painting reconstruction network and its parameters are loaded into the memory for running.

[0148] In step 402, target elements that are not suitable for the oil painting style in volume are detected in original image data to be reconstructed.

[0149] ​Generally, the structure of the oil painting reconstruction network is large and occupies more resources, and is usually deployed on the server. The server can encapsulate the oil painting reconstruction network into an interface, a plug-in, etc., and provide a service of reconstructing the oil painting style to users in a local area network or a public network. The user can transmit image data of a to-be-reconstructed oil painting style to the server by calling the interface, the plug-in, etc. through a client or a browser for the convenience of distinction. The image data of the to-be-reconstructed style is referred to as original image data.

[0150] Since the oil painting is mostly blurred, the details in the original image data are easily blurred, which may affect the business. Therefore, the server can perform target detection in the original image data. The target elements that are not suitable for the oil painting style in size usually are small-volume data, and part of the information may be lost due to blurring in the reconstruction of the oil painting style.

[0151] Of course, if the local resources of an electronic device such as a personal computer, a notebook computer, etc. are sufficient to meet the operation of the oil painting reconstruction network, the oil painting reconstruction network can be loaded and run locally on the electronic device. At this time, the original image data of the to-be-reconstructed oil painting style can be input through a command line, etc., and the target elements that are not suitable for the oil painting style in size can be detected in the original image data.

[0152] In one example, the target element is face data. In this example, face detection can be performed on the original image data to be reconstructed to obtain face data that is not suitable for the oil painting style in size.

[0153] Face detection, also known as face key point detection, positioning or face alignment, refers to locating the key region position of a face in given face data, including eyebrows, eyes, nose, mouth, face contour, etc.

[0154] Face detection usually uses the following methods:

[0155] 1. Artificially extracted features, such as haar features, are used to train a classifier, and the classifier is used for face detection.

[0156] 2. Inherit face detection from a general target detection algorithm, for example, use Faster R-CNN to detect faces.

[0157] 3. Use a convolutional neural network with a cascade structure, for example, Cascade CNN (Cascade Convolutional Neural Network) and MTCNN (Multi-task Cascaded Convolutional Networks).

[0158] In the present example, the algorithm for face detection has low requirements, simply marks the face data, allows certain noise to exist, and a general algorithm such as dlib can be used for face detection to obtain 68 face key points, which can meet the demand of the business.

[0159] In another example, the target element is subtitle data, and in the present example, optical character recognition (OCR) is performed on the original image data to be reconstructed to obtain subtitle data that is not suitable in volume for the oil painting style.

[0160] On the one hand, the network for implementing optical character recognition can be TexNet (text network), HRNet (High-Resolution Net), etc., and can be trained using language corpus of languages used in regions related to the business (such as Chinese, English, etc.).

[0161] On the other hand, the region where the subtitle is located is generally located at the bottom of the original image data, and the text information in the specified region in the text information obtained by OCR recognition is considered to be the subtitle, allowing certain noise to exist.

[0162] Of course, the above target element is only an example, and other target elements can be set according to actual conditions when implementing the present embodiment, and the present embodiment does not limit this. In addition, in addition to the above target element, other target elements can be used according to actual needs by those skilled in the art, and the present embodiment does not limit this.

[0163] Step 403, inputting the original image data into the oil painting reconstruction network to reconstruct candidate image data containing an oil painting style.

[0164] In the present embodiment, the original image data is input into the oil painting reconstruction network, and the oil painting reconstruction network processes the original image data according to its structure to reconstruct the original image data into new image data containing an oil painting style, which is denoted as candidate image data.

[0165] Step 404, pasting the target element back into the candidate image data to obtain target image data.

[0166] In actual application, the candidate image data can cause information loss of the target element to a certain extent, and therefore each target element can be pasted back to the original position in the candidate image data and appropriately processed to obtain the final target image data.

[0167] In one example, if the target element is face data, the face key points marked when detecting the face data can be queried, and the face data is mapped back to the candidate image data through the face key points, that is, each face key point is traversed, and for the same face key point, the pixel point representing the face key point in the face data is aligned with the pixel point representing the face key point in the candidate image data, and when the alignment is completed, the face data is mapped on the candidate image data.

[0168] In order to improve the fusion degree of the face data and the surrounding, the face data can be subjected to blur processing (such as Gaussian blur) to obtain target image data.

[0169] For example, for the original image data as shown in Figure 5A , on the one hand, the face data is extracted, and on the other hand, the oil painting reconstruction network is used to reconstruct the candidate image data as shown in Figure 5B , compared with the original image data as shown in Figure 5B , whether it is face data or background (such as the lines of the window, the characters of the curtain, etc.), there is a certain blur due to the oil painting style, at this time, as shown in Figure 5A , the face data of the original image data is pasted back to the candidate image data to obtain target image data, and the face data of the target image data is relatively clear, and the background conforms to the oil painting style. Figure 5C

[0170] In another example, if the target element is subtitle data, the position marked when detecting the subtitle data can be queried, and then the subtitle data can be mapped back to the candidate image data through the position to obtain target image data, that is, each pixel point of the subtitle data is traversed, and the position of the pixel point in the original image data is queried, and the pixel point at the same position in the candidate image data is replaced.

[0171] Of course, the above-mentioned way of pasting the target element back to the candidate image data is only as an example, and when implementing the present embodiment, other ways of pasting the target element back to the candidate image data can be set according to actual conditions, and the present embodiment does not limit this. In addition, in addition to the above-mentioned way of pasting the target element back to the candidate image data, other ways of pasting the target element back to the candidate image data can also be used by those skilled in the art according to actual needs, and the present embodiment does not limit this.

[0172] Thereafter, the target image data can be subjected to overall image processing, for example, adjusting brightness, adjusting saturation, etc., to improve the quality of the target image data.

[0173] ​In the embodiment, the oil painting reconstruction network is loaded; target elements that are not suitable for the oil painting style in volume are detected in original image data to be reconstructed; the original image data is input into the oil painting reconstruction network to be reconstructed into candidate image data containing the oil painting style; the target elements are pasted back into the candidate image data to obtain target image data. The oil painting reconstruction network is trained based on samples balancing oil painting styles of different eras, has balanced performance, avoids problems caused by a single era of oil painting style, and belongs to a generative adversarial network. The oil painting reconstruction network can reconstruct image data into the oil painting style, and the reconstruction of the oil painting style belongs to post-processing, which can maintain the threshold of producing video data and the time consumption of producing video data, and greatly improves the efficiency of producing video data of the oil painting style. Moreover, some target elements that are not suitable for the reconstruction of the oil painting style are directly migrated from the original image data before reconstruction to the target image data after reconstruction, which can improve the quality of the target image data and adapt to requirements of various businesses.

[0174] Embodiment Four

[0175] Figure 6 A flowchart of a video reconstruction method provided by Embodiment Four of the application. The embodiment can be applied to the case of reconstructing video data into the oil painting style based on the oil painting reconstruction network. The method can be executed by a video reconstruction device, which can be realized in the form of hardware and / or software and can be configured in an electronic device. As shown in the figure, the method comprises the following steps. Figure 6

[0176] Step 601: Load the oil painting reconstruction network.

[0177] In a specific implementation, the oil painting reconstruction network can be trained in advance according to the method described in Embodiment One of the application, wherein the oil painting reconstruction network is used to reconstruct image data of a non-oil painting style into image data of an oil painting style.

[0178] When the oil painting reconstruction network is applied, the oil painting reconstruction network and its parameters are loaded into the memory for running.

[0179] Step 602: Obtain original video data of which the content is to introduce a game.

[0180] In the embodiment, an art personnel can produce video data for a game to be promoted, and the content of the video data is to introduce the game.

[0181] The type of the game can include a Multiplayer Online Battle Arena (MOBA), a Role-playing game (RPG), a Simulation Game (SLG), and the like, and the embodiment does not limit the type of the game.​

[0182] In a specific implementation, the content of the original video data can be divided into two main forms of game content and real plot, and the plot can be further divided into the following categories:

[0183] 1. Pseudo food sharing

[0184] The original video data contains some food-related materials that can attract the attention of users. Secondly, the play of earning money by eating food is implanted, and at the same time, a clear goal of playing the game is provided for users.

[0185] 2. Close to user life theme

[0186] The original video data is close to the current life state of the user, and the selling point of the game is implanted in all aspects of life. The target game props are purchased, and the game is used to earn money for payment by eating and buying snacks. The production of such materials is relatively simple, the scene is single, and the shooting difficulty is low. The first half of the material is mainly two-person dialogue, and the second half is the game implant segment.

[0187] 3. Situation drama

[0188] The original video data contains situation drama materials, and some cases are star endorsements wearing game costumes, and some plots are exaggerated to attract the attention of users.

[0189] Generally, the structure of the oil painting reconstruction network is relatively large and occupies more resources, and is usually deployed on the server. The server can encapsulate the oil painting reconstruction network into interfaces, plugins, etc., and provide services for reconstructing the oil painting style to users in the local area network or public network. Users can call the interface, plugin, etc. through the client or browser to transmit the image data to be reconstructed to the server. For convenience, the image data to be reconstructed is referred to as original image data.

[0190] Since oil paintings mostly have blurred conditions, details in the original image data can be easily blurred, which may affect business. Therefore, the server can perform target detection in the original image data. Target elements that are not suitable in volume for the oil painting style. These target elements are usually small-volume data, which may lose some information due to blurring in the reconstruction of the oil painting style.

[0191] Of course, if the local resources of electronic devices such as personal computers, notebook computers, etc. are sufficient to meet the operation of the oil painting reconstruction network, the oil painting reconstruction network can be loaded and run locally on the electronic device. At this time, the original image data to be reconstructed in the oil painting style can be input through the command line, and target elements that are not suitable in volume for the oil painting style can be detected in the original image data.

[0192] The original video data has multiple frames of original image data, and the contents in different frames of original image data have certain differences, and at least part of the original image data has target elements that are not suitable in volume for the oil painting style.

[0193] In one example, the target element is face data, and in this example, face detection can be performed on the original image data to be reconstructed to obtain face data that is not suitable in volume for the oil painting style.

[0194] Face detection, also known as face key point detection, positioning, or face alignment, refers to locating the key area positions of a face, including eyebrows, eyes, nose, mouth, face contour, etc.

[0195] Face detection usually uses the following methods:

[0196] 1. Use artificial feature extraction, such as Haar features, use feature training classifiers, and use classifiers for face detection.

[0197] 2. Inherit face detection from general object detection algorithms, for example, use Faster R-CNN to detect faces.

[0198] 3. Use a convolutional neural network with a cascade structure, such as Cascade CNN and MTCNN.

[0199] In this example, the algorithm for face detection has low requirements, simply marks the face data, allows certain noise, and can use general algorithms such as dlib for face detection to obtain 68 face key points, which can meet the business requirements.

[0200] In another example, the target element is subtitle data, and in this example, optical character recognition is performed on the original image data to be reconstructed to obtain subtitle data that is not suitable in volume for the oil painting style.

[0201] On the one hand, the network that implements optical character recognition can be TexNet (text network), HRNet (High-Resolution Net), etc., and can be trained using language corpora used in regions related to the business, such as Chinese and English.

[0202] On the other hand, the region where the subtitles are located is generally located at the bottom of the original image data, and the text information in the specified region in the OCR-recognized text information is considered to be subtitles, allowing certain noise.

[0203] Of course, the above target elements are only examples, and other target elements can be set according to actual conditions in implementing the embodiment, and the embodiment does not limit this. In addition, in addition to the above target elements, other target elements can be used according to actual needs by those skilled in the art, and the embodiment does not limit this.

[0204] Further, in order to smooth the target image data of each frame later, in a case where the original face data is recognized but the face key point is not recognized, the detection frame of the current frame original image data can be compared with the detection frame of the previous frame image data, and in a case where they are similar, the face key point detected in the previous frame can be reused.

[0205] In a case where the original face data is not recognized, the change between the face region of the current frame original image data and the face region of the previous frame image data can be compared, and if the change is small (expressed as being less than a change threshold), the recognition result (i.e., the face key point and the detection frame) of the previous frame can be reused.

[0206] Step 603, inputting the original image data into the oil painting reconstruction network to reconstruct candidate image data containing an oil painting style.

[0207] In the embodiment, the original image data is input into the oil painting reconstruction network, and the oil painting reconstruction network processes the original image data according to its structure to reconstruct new image data containing an oil painting style, which is recorded as candidate image data.

[0208] Step 604, pasting the target elements back into the candidate image data to obtain target image data.

[0209] In actual application, the candidate image data can cause information loss of the target elements to a certain extent, and therefore each target element can be pasted back into the original position in the candidate image data and appropriately processed to obtain the final target image data.

[0210] In one example, if the target element is face data, the face key point marked when detecting the face data can be queried, and the face data is mapped back into the candidate image data through the face key point, that is, each face key point is traversed, the pixel point representing the face key point in the face data is aligned with the pixel point representing the face key point in the candidate image data for the same face key point, and when the alignment is completed, the face data is pasted on the candidate image data.

[0211] In order to improve the fusion degree of the face data and the surrounding, the face data can be subjected to blur processing (such as Gaussian blur) to obtain the target image data.

[0212] In another example, if the target element is subtitle data, the position marked when detecting the subtitle data can be queried, and the subtitle data can be mapped back to the candidate image data through the position to obtain the target image data, that is, each pixel point of the subtitle data is queried for its position in the original image data, and the pixel point is replaced with a pixel point at the same position in the candidate image data.

[0213] Of course, the above method of pasting the target element back to the candidate image data is only an example, and other methods of pasting the target element back to the candidate image data can be set according to actual conditions when implementing the present embodiment, and the present embodiment does not limit this. In addition, in addition to the above method of pasting the target element back to the candidate image data, other methods of pasting the target element back to the candidate image data can also be used by those skilled in the art according to actual needs, and the present embodiment does not limit this.

[0214] Thereafter, overall image processing can be performed on the target image data, for example, adjusting brightness, adjusting saturation, etc., to improve the quality of the target image data.

[0215] Step 605, replacing the target image data with the original image data in the original video data to obtain target video data.

[0216] In the original video data, the target image data can be replaced with the corresponding original image data to obtain target video data.

[0217] Thereafter, game-related advertising element data can be added to the target video data to obtain advertising video data, wherein the advertising element data includes a LOGO (icon) of a platform for distributing the target game, a Banner (banner advertisement), an EC (ending segment, generally containing information of the target game such as name, platform for distributing the target game, etc.), etc.

[0218] The advertising video data is published in a specified channel (such as news information, short video, novel reading, sports health, etc.), so that when the client accesses the channel, the advertising video data is pushed to the client for playing, and the user downloads the game from the platform for distributing the target game when interested in the target game.

[0219] In the embodiment, the oil painting reconstruction network is loaded; original video data introducing a game is acquired, the original video data has multiple frames of original image data, at least part of the original image data has a target element that is not suitable in volume for an oil painting style; the original image data is input into the oil painting reconstruction network to be reconstructed into candidate image data containing an oil painting style; the target element is pasted back into the candidate image data to obtain target image data; the target image data is replaced with the original image data in the original video data to obtain target video data. The oil painting reconstruction network is trained based on samples balancing oil painting styles of different eras, has balanced performance, avoids problems caused by a single era of oil painting style, and belongs to a generative adversarial network. The oil painting reconstruction network can reconstruct image data into an oil painting style, the reconstruction of the oil painting style belongs to post-processing, can maintain a threshold for producing video data, maintains time consumption for producing video data, and greatly improves efficiency of producing video data in an oil painting style. Moreover, some target elements that are not suitable for being reconstructed into an oil painting style are directly migrated from original image data before reconstruction to target image data after reconstruction, which can improve quality of the target image data and adapt to requirements of various businesses.

[0220] Embodiment five

[0221] Figure 7 A structural schematic diagram of a training device of an oil painting reconstruction network provided by the embodiment five of the application is shown in FIG. 5. Figure 7 As shown in the figure, the device comprises:

[0222] A first sample image data acquisition module 701 is configured to acquire first sample image data of a real scene;

[0223] A second sample image data acquisition module 702 is configured to acquire second sample image data of an oil painting;

[0224] A second sample image data addition module 703 is configured to add new second sample image data to balance oil painting styles of different eras if the second sample image data converges to an oil painting style of a certain era;

[0225] A generative adversarial network training module 704 is configured to train a generative adversarial network with the first sample image data as a source of oil painting content and the second sample image data as a source of oil painting style to obtain an oil painting reconstruction network.

[0226] In an embodiment of the application, the second sample image data acquisition module 702 is further configured to:

[0227] acquire one or more oil painting data sets in which multiple frames of image data of an oil painting are stored;

[0228] determine a painter;

[0229] Filtering, in the oil painting data set, image data of which the content is an oil painting drawn by the painter, as second image data.

[0230] In an embodiment of the present application, the second sample image data adding module 703 is further configured to:

[0231] Inquire the painter who draws the oil painting represented by the second sample image data;

[0232] If the time when the painter draws the oil painting converges to a certain era, perform stylization processing on the first sample image data, so that the style of the first sample image data is converted to an oil painting style;

[0233] Add the first sample image data with the oil painting style as new second sample image data.

[0234] In an embodiment of the present application, the second sample image data adding module 703 is further configured to:

[0235] Convert the first sample image data to grayscale image data;

[0236] Normalize the original grayscale value of the grayscale image data to a specified range to obtain a candidate grayscale value;

[0237] Add a window to the grayscale image data to move the window to the grayscale image data as a target;

[0238] In each movement of the window, filter the candidate grayscale value with the largest number in the window as a target grayscale value;

[0239] Average the color value of the target pixel point in the first sample image data, the target pixel point being a pixel point located at the position of the target grayscale value;

[0240] Replace the color value of the target pixel point with the average value.

[0241] In an embodiment of the present application, the second sample image data adding module 703 is further configured to:

[0242] Collect unlabeled image data as unknown image data;

[0243] Load multiple classifiers for binary classification;

[0244] Call multiple classifiers in turn to label the unknown image data with binary classification labels;

[0245] Add the unknown image data with the label of modern realistic oil painting as new second sample image data.

[0246] In an embodiment of the present application, the classifier comprises a first classifier, a second classifier, a third classifier; the second sample image data adding module 703 is further used for:

[0247] calling the first classifier to label whether the content of the unknown image data is a painting;

[0248] if the label of the unknown image data includes a painting, calling the second classifier to label whether the content of the unknown image data is realistic;

[0249] if the label of the unknown image data includes a realistic painting, calling the third classifier to label whether the content of the unknown image data is modern.

[0250] In an embodiment of the present application, the first classifier has a first feature extraction layer and a first mapping layer, the second classifier has a second feature extraction layer and a second mapping layer, and the third classifier has a third feature extraction layer and a third mapping layer;

[0251] The second sample image data adding module 703 is further used for:

[0252] inputting the unknown image data into the first feature extraction layer to extract a first image feature;

[0253] inputting the first image feature into the first mapping layer to map a label of whether the content of the unknown image data is a painting;

[0254] The second sample image data adding module 703 is further used for:

[0255] inputting the first image feature into the second feature extraction layer to extract a second image feature;

[0256] inputting the second image feature into the second mapping layer to map a label of whether the content of the unknown image data is realistic;

[0257] The second sample image data adding module 703 is further used for:

[0258] inputting the second image feature into the third feature extraction layer to extract a third image feature;

[0259] inputting the third image feature into the third mapping layer to map a label of whether the content of the unknown image data is modern.

[0260] In an embodiment of the present application, further comprising:

[0261] a seed image data marking module, configured to mark the second sample image data as seed image data if the oil painting represented by the second sample image data is authentic and typical;

[0262] a vividness calculation module, configured to respectively calculate the vividness of the first sample image data and the seed image data in color;

[0263] a threshold value construction module, configured to construct a threshold value based on the vividness of the seed image data in color;

[0264] a sample image data filtering module, configured to filter the first sample image data if the vividness of the first sample image data in color is less than the threshold value.

[0265] In an embodiment of the present application, the vividness calculation module is further configured to:

[0266] respectively divide the first sample image data and the seed image data in color into red vectors, green vectors and blue vectors;

[0267] take the absolute value of the difference between the red vectors and the green vectors to obtain red-green vectors;

[0268] take the absolute value of the difference between the mixed vectors and the blue vectors to obtain yellow-blue vectors, wherein the mixed vectors are the sum of the red vectors and the green vectors multiplied by a specified first proportion;

[0269] calculate the first standard deviation and the first average value of the red-green vectors;

[0270] calculate the second standard deviation and the second average value of the yellow-blue vectors;

[0271] take the square root of the sum of the square of the first standard deviation and the square of the second standard deviation to obtain a third standard deviation;

[0272] calculate the square root of the sum of the square of the first average value and the square of the second average value to obtain a third average value;

[0273] take the second proportion of the third average value and add it to the third standard deviation to obtain the vividness.

[0274] The training device of the oil painting reconstruction network provided in the embodiments of the present application can execute the training method of the oil painting reconstruction network provided in any embodiment of the present application, and has the corresponding function modules and beneficial effects of executing the training method of the oil painting reconstruction network.

[0275] Embodiment six

[0276] Figure 8A structural schematic diagram of an image reconstruction device provided by Embodiment Six of the present application is shown in FIG. 8. As shown in the figure, the device comprises: Figure 8

[0277] An oil painting reconstruction network loading module 801 is configured to load an oil painting reconstruction network trained according to the method of any of the embodiments of the present application.

[0278] A target element detection module 802 is configured to detect target elements that are not suitable for an oil painting style in volume in original image data to be reconstructed.

[0279] A candidate image data reconstruction module 803 is configured to input the original image data into the oil painting reconstruction network to reconstruct the original image data into candidate image data containing an oil painting style.

[0280] A target image data generation module 804 is configured to paste the target elements back into the candidate image data to obtain target image data.

[0281] In an embodiment of the present application, the target element detection module 802 is further configured to:

[0282] perform face detection in the original image data to be reconstructed to obtain face data that is not suitable for an oil painting style in volume;

[0283] and / or

[0284] perform optical character recognition in the original image data to be reconstructed to obtain subtitle data that is not suitable for an oil painting style in volume.

[0285] In an embodiment of the present application, the target image data generation module 804 is further configured to:

[0286] query face key points marked when the face data is detected;

[0287] map the face data back into the candidate image data through the face key points;

[0288] perform blur processing on the face data to obtain target image data;

[0289] and / or

[0290] query positions marked when the subtitle data is detected;

[0291] map the subtitle data back into the candidate image data through the positions to obtain target image data.

[0292] The image reconstruction device provided by the embodiments of the present application can execute the image reconstruction method provided by any of the embodiments of the present application and has the corresponding functional modules and beneficial effects of executing the image reconstruction method. ​

[0293] Embodiment seven

[0294] Figure 9 A structural schematic diagram of a video reconstruction device provided by an embodiment of the present application is shown in FIG. 7. As shown in the figure, the device comprises: Figure 9

[0295] An oil painting reconstruction network loading module 901, configured to load an oil painting reconstruction network trained according to the method of any embodiment of the present application;

[0296] An original video data acquisition module 902, configured to acquire original video data with content of introducing a game, wherein the original video data comprises a plurality of original image data, and at least part of the original image data comprises a target element that is not suitable in volume for an oil painting style;

[0297] A candidate image data reconstruction module 903, configured to input the original image data into the oil painting reconstruction network to reconstruct the original image data into candidate image data containing an oil painting style;

[0298] A target image data generation module 904, configured to paste the target element back into the candidate image data to obtain target image data;

[0299] A target video data generation module 905, configured to replace the original image data with the target image data in the original video data to obtain target video data.

[0300] In an embodiment of the present application, the device further comprises:

[0301] An advertisement video data generation module, configured to add an advertisement element related to the game in the target video data to obtain advertisement video data;

[0302] An advertisement video data publishing module, configured to publish the advertisement video data in a designated channel to push the advertisement video data to a client for playing when the client accesses the channel.

[0303] The video reconstruction device provided by the embodiment of the present application can execute the video reconstruction method provided by any embodiment of the present application, and has the corresponding function modules and beneficial effects of executing the video reconstruction method.

[0304] Embodiment eight

[0305] Figure 10 ​A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0306] like Figure 10 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0307] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0308] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the training methods for oil painting reconstruction networks, image reconstruction methods, and video reconstruction methods.

[0309] In some embodiments, the training method of the oil painting reconstruction network, the image reconstruction method, the video reconstruction method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded onto and / or installed in the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the training method of the oil painting reconstruction network, the image reconstruction method, the video reconstruction method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the training method of the oil painting reconstruction network, the image reconstruction method, the video reconstruction method by any other appropriate means (for example, by means of firmware).

[0310] The various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0311] Computer programs used to implement the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program running on the processor implements the functions / operations specified in the flowcharts and / or the block diagrams. The computer program can be executed entirely on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0312] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0313] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0314] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), blockchain network, and the Internet.

[0315] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0316] It should be understood that the various forms of flow shown above can be reordered, added to, or have steps deleted. For example, the steps described in the present application can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which are not limited herein.

[0317] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A training method for an oil painting reconstruction network, characterized in that, include: Acquire first sample image data of real-world scenes; Obtain second sample image data containing oil paintings; If the second sample image data converges to the oil painting style of a certain era, then new second sample image data is added to balance the oil painting styles of different eras. A generative adversarial network is trained using the first sample image data as the source of the oil painting content and the second sample image data as the source of the oil painting style to obtain an oil painting reconstruction network. If the oil painting represented by the second sample image data is authentic and typical, then the second sample image data will be marked as seed image data. The vividness of colors in the first sample image data and the seed image data are calculated respectively; A threshold is constructed based on the vividness of the colors in the seed image data; If the vividness of the first sample image data in terms of color is less than the threshold, then the first sample image data is filtered out. Wherein, if the second sample image data converges to the oil painting style of a certain era, new second sample image data is added to balance the oil painting styles of different eras, including: Query the painter who created the oil painting represented by the second sample image data; If the time when the painter created the oil painting converges to a certain era, then the first sample image data is stylized to convert the style of the first sample image data into the style of an oil painting. Add the first sample image data with an oil painting style as the new second sample image data; or, Collect unlabeled image data as unknown image data; Load multiple classifiers for binary classification; The classifiers are called sequentially to label the unknown image data with binary classification labels; Add the unknown image data labeled as modern realist oil painting as new second sample image data; The step of performing stylization processing on the first sample image data to convert the first sample image data into an oil painting style includes: Convert the first sample image data into grayscale image data; The original grayscale values ​​of the grayscale image data are normalized to a specified range to obtain candidate grayscale values; A window is added to the grayscale image data, and the window is moved with the goal of traversing the grayscale image data; Each time the window is moved, the candidate grayscale value with the largest number of occurrences in the window is selected as the target grayscale value. In the first sample image data, the average color value of the target pixel is calculated, where the target pixel is the pixel located at the position of the target gray value; Replace the color value of the target pixel with the average value.

2. The method according to claim 1, characterized in that, The acquired content is the second sample image data of the oil painting, including: Obtain one or more oil painting datasets, wherein the oil painting datasets store image data of multiple frames containing oil paintings; Select the painter; Image data containing oil paintings by the painter are selected from the oil painting dataset and used as the second image data.

3. The method according to claim 1, characterized in that, The classifier includes a first classifier, a second classifier, and a third classifier; The step of sequentially calling multiple classifiers to label the unknown image data with binary classification labels includes: The first classifier is invoked to label whether the content of the unknown image data is an oil painting; If the label of the unknown image data includes oil painting, then the second classifier is called to determine whether the label of the unknown image data is a realistic label; If the label of the unknown image data includes realistic oil painting, then the third classifier is invoked to determine whether the content of the unknown image data is labeled as modern.

4. The method according to claim 3, characterized in that, The first classifier has a first feature extraction layer and a first mapping layer, the second classifier has a second feature extraction layer and a second mapping layer, and the third classifier has a third feature extraction layer and a third mapping layer; The step of calling the first classifier to label the unknown image data to determine whether the content is an oil painting includes: The unknown image data is input into the first feature extraction layer to extract the first image features; Input the first image feature into the first mapping layer to map whether the content of the unknown image data is a label for an oil painting; The step of calling the second classifier to label the unknown image data as whether the content is realistic includes: The first image features are input into the second feature extraction layer to extract the second image features; The second image feature is input into the second mapping layer to map whether the content of the unknown image data is a realistic label; The step of calling the third classifier to label the unknown image data as modern includes: The second image features are input into the third feature extraction layer to extract the third image features; The third image feature is input into the third mapping layer to map whether the content of the unknown image data is a modern label.

5. The method according to claim 1, characterized in that, The step of separately calculating the vividness of colors in the first sample image data and the seed image data includes: The first sample image data and the seed image data are respectively divided into red vector, green vector, and blue vector based on color. The absolute value of the difference between the red vector and the green vector is taken to obtain the red-green vector; The absolute value of the difference between the blended vector and the blue vector is taken to obtain the yellow-blue vector, wherein the blended vector is a specified first ratio of the sum of the red vector and the green vector; Calculate the first standard deviation and the first mean of the red-green vector; Calculate the second standard deviation and second mean of the yellow-blue vector; The third standard deviation is obtained by taking the square root of the sum of the squares of the first standard deviation and the squares of the second standard deviation. Calculate the square root of the sum of the squares of the first average and the squares of the second average to obtain the third average; The third average value is taken as a specified second proportion and added to the third standard deviation to obtain the vividness.

6. An image reconstruction method, characterized in that, include: Load the oil painting reconstruction network trained by the method according to any one of claims 1-5; Detect target elements in the original image data to be reconstructed that do not fit the style of oil painting in terms of volume; The original image data is input into the oil painting reconstruction network and reconstructed into candidate image data containing oil painting style; The target element is pasted back into the candidate image data to obtain the target image data.

7. The method according to claim 6, characterized in that, The detection of target elements in the original image data to be reconstructed that do not conform to the oil painting style in terms of volume includes: Face detection is performed on the original image data to be reconstructed, resulting in face data whose volume does not match the style of oil painting; and / or Optical character recognition is performed on the original image data to be reconstructed to obtain subtitle data that is not suitable for the style of oil painting in terms of volume.

8. The method according to claim 7, characterized in that, The step of pasting the target element back into the candidate image data to obtain the target image data includes: Query the facial landmarks marked during the detection of the facial data; The facial data is mapped back to the candidate image data using the facial key points; The face data is blurred to obtain the target image data; and / or Query the position marked when the subtitle data was detected; The target image data is obtained by mapping the caption data back to the candidate image data using the location.

9. A video reconstruction method, characterized in that, include: Load the oil painting reconstruction network trained by the method according to any one of claims 1-5; The content obtained is the original video data introducing the game, which contains multiple frames of original image data, and at least some of the original image data contains target elements whose volume does not match the oil painting style. The original image data is input into the oil painting reconstruction network and reconstructed into candidate image data containing oil painting style; The target element is pasted back into the candidate image data to obtain the target image data; The target image data is replaced in the original video data to obtain the target video data.

10. The method according to claim 9, characterized in that, Also includes: Add game-related advertising elements to the target video data to obtain advertising video data; The advertising video data is published on a designated channel so that when a client accesses the channel, the advertising video data is pushed to the client for playback.

11. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the training method of the oil painting reconstruction network according to any one of claims 1-5, or the image reconstruction method according to any one of claims 6-8, or the video reconstruction method according to any one of claims 9-10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the training method of the oil painting reconstruction network according to any one of claims 1-5, the image reconstruction method according to any one of claims 6-8, or the video reconstruction method according to any one of claims 9-10.

Images