Video generation method for consistent style transfer of electronic photo walls

Abstract

The present invention discloses a video generation method for style migration consistency of electronic photo walls, belonging to the field of computer vision and artificial intelligence technology. The method of the present invention first obtains the optimal style migration image of the first frame, and then iteratively generates the optimal style migration image frame by frame. Specifically, based on the optimal style migration image of the t-th frame image, the optimal style migration image of the t+1-th frame image is obtained under the condition of minimizing the loss function of the image style migration module, wherein the loss function of the image style migration module includes the loss of content inconsistency index and illumination inconsistency index. The method solves the two problems of content inconsistency and illumination inconsistency of previous and next frames in the style migration process at a relatively low computational cost.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence technology, and in particular relates to a video generation method for consistency in electronic photo wall style migration. Background Art

[0002] The electronic photo wall is a new product composed of multiple LCD screens that can display different content, such as images and videos, individually or in combination. Users can capture videos, transfer their styles, and then place them on the electronic photo wall, increasing human-computer interaction and making the content more engaging. For example, an ordinary group dance video can be transferred to the style of a model opera, or a video of a children's dance performance can be enhanced with animal hair accessories.

[0003] However, existing video style transfer algorithms primarily include image-to-image conversion methods, temporal modeling and consistency optimization methods, and pose-based constraint methods. Image-to-image conversion methods primarily draw on techniques for style transfer from static images, performing style transfer on each frame individually. Key to this is the use of deep neural networks to learn the transformation between the source and target style images. While effective, these methods are prone to temporal inconsistencies. Summary of the Invention

[0004] The technical problem to be solved by the present invention is that existing video style transfer methods have the problem of low quality of generated videos due to timing inconsistency. The present invention proposes a video generation method for style transfer consistency of electronic photo walls. Based on existing artificial intelligence and electronic display technology, the method is targeted at the specific objects of electronic photo walls and, under limited computing power, controls the timing inconsistency of the previous and next frames of the video within a certain acceptable range.

[0005] The video generation method for consistent style transfer of electronic photo walls of the present invention obtains the optimal style transfer image Itran(t,op) of the t+1th frame image based on the optimal style transfer image Itran(t,op) of the tth frame image, where t≥1. The specific process is:

[0006] Use the image style transfer module to perform style transfer on the t+1th frame image in the video, and obtain rIax candidate style transfer images Itran(t+1,ri), where ri is the index of the candidate style transfer image, ranging from 1 to rIax; calculate the inconsistency index between the optimal style transfer image Itran(t,op) of the tth frame image and the candidate style transfer image Itran(t+1,ri);

[0007] Adding the inconsistency indicator as an inconsistency loss to the loss function of the image style transfer module to obtain an updated loss function;

[0008] Based on the principle of minimizing the updated loss function, the optimal style transfer image Itran(t+1,op) of the t+1th frame image is selected from the rIax candidate style transfer images Itran(t+1,ri).

[0009] Based on the method of the present invention for obtaining the optimal style transfer image Itran(t,op) of the t+1th frame image from the optimal style transfer image Itran(t,op) of the tth frame image, the optimal style transfer images of all images in the video are calculated in sequence to obtain a video after style transfer.

[0010] Furthermore, the inconsistency index includes a content inconsistency index; the content inconsistency index is calculated as follows:

[0011] Input the optimal style transfer image Itran(t,op) of the t-th frame image and the candidate style transfer image Itran(t+1,ri) into the previous and next frame content inconsistency assessment module, obtain the difference map CM between the image Itran(t,op) and the image Itran(t+1,ri), and count the number of pixels Pcm in the difference map CM;

[0012] calculating a difference level DB based on the difference map CM;

[0013] The inconsistency index Dd is expressed as:

[0014] Dd(t,op;t+1,ri)=Pcm(t,op;t+1,ri)*DB(t,op;t+1,ri);

[0015] The inconsistency indicator Dd is used as the content inconsistency loss Added to the loss function of the image style transfer module.

[0016] Furthermore, the previous and next frame content inconsistency assessment module adopts a pruned MGCDT structure, including a Siamese dual-branch encoder, a multi-granularity feature fusion module, and a deep feature guidance unit;

[0017] The Siamese dual-branch encoder is used to extract the multi-scale features of the input image Itran(t,op) and the multi-scale features of the image Itran(t+1,ri), respectively, and then input them into the multi-granularity feature fusion module to generate a difference feature map, and the difference feature map is input into the deep feature guidance unit for feature enhancement to generate a difference map CM.

[0018] Furthermore, the inconsistency index also includes an illumination inconsistency index; the illumination inconsistency index is calculated as follows:

[0019] The difference map CM output by the previous and next frame content inconsistency assessment module includes the difference area DA of the previous and next frames. Based on the difference area DA, difference area images IDAorg(t) and IDAorg(t+1) are extracted from the image Itran(t,op) and the image Itran(t+1,ri), respectively, and indifference area images InonDAorg(t) and InonDAorg(t+1) are extracted from the image Itran(t,op) and the image Itran(t+1,ri), respectively.

[0020] For the difference area images IDAorg(t) and IDAorg(t+1), the optical flow method is used to calculate the optical flows FlowIDAorg(t) and FlowIDAorg(t+1);

[0021] The cosine theorem formula of the vector space model is used to calculate the relationship between each pixel corresponding to the optical flow FlowIDAorg(t) and FlowIDAorg(t+1), and the optical flow change feature matrix SFIDAorg(t,t+1) of the difference area is obtained;

[0022] Calculate the average value avgSFIDAorg(t,t+1) and variance sigSFIDAorg(t,t+1) of the optical flow change feature matrix SFIDAorg(t,t+1);

[0023] Calculate the optical flow change feature matrix SFInonDAorg(t,t+1) of the indifference area images InonDAorg(t) and InonDAorg(t+1), and calculate the average value avgSFInonDAorg(t,t+1) and variance sigSFInonDAorg(t,t+1) of the optical flow change feature matrix SFInonDAorg(t,t+1);

[0024] Constructing an illumination inconsistency metric as a loss of optical flow variation Added to the loss function of the image style transfer module, the loss of optical flow change It is expressed as follows:

[0025] .

[0026] Furthermore, the method for obtaining the optimal style transfer image Itran(t,op) of the first frame image is as follows:

[0027] The image style transfer module is used to transfer the style of the first frame image in the video, and rIax candidate style transfer images are obtained, so that the loss function of the style transfer module Based on the minimum principle, the optimal style transfer image Itran(1,op) of the first frame image is selected from the rIax candidate style transfer images.

[0028] Furthermore, the inconsistency index is added as the inconsistency loss to the loss function of the image style transfer module to obtain the updated loss function , expressed as

[0029]

[0030] in, 、 、 、 is a hyperparameter, Represents resistance to loss, represents the cycle consistency loss, Indicates content inconsistency loss, Represents the loss of optical flow change.

[0031] Furthermore, the backbone network of the image style transfer module is CycleGAN.

[0032] Furthermore, the optical flow change feature matrix SFInonDAorg(t,t+1) of the indifference area images InonDAorg(t) and InonDAorg(t+1) is calculated. Specifically, no more than 10 points are randomly selected, and the optical flow method is used to calculate the optical flows FlowInonDAorg(t) and FlowInonDAorg(t+1). The vector space model VSM method is also used to calculate the optical flow change feature matrix SFInonDAorg(t,t+1) of the indifference area of ​​all points.

[0033] Furthermore, the calculation of the difference level DB based on the difference map CM is specifically as follows:

[0034] For each pixel in the difference map CM, the difference value of the feature vector at the corresponding position of the image Itran(t,op) and the image Itran(t+1,ri) is calculated, and the difference values ​​are averaged to obtain the difference level DB.

[0035] Beneficial Effects: Classic transfer algorithms are only suitable for single-frame image transfer. When applied to video style transfer, they often encounter inconsistent content and illumination between previous and subsequent frames. This method, by introducing a content inconsistency assessment module (DEM) and an illumination inconsistency assessment module (DFM), combined with the CycleGAN framework, achieves temporal consistency control during video style transfer at a low computational cost, thereby generating temporally coherent and stylistically stable video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is an overall flow chart of the method of the present invention.

[0037] Figure 2 Schematic diagram of the relationship between the various structures in the image style transfer module of the present invention. DETAILED DESCRIPTION

[0038] The present invention is directed to a video generation method for electronic photo wall style migration consistency, such as Figure 1 As shown, the following steps are included:

[0039] Step 1: extract the original image sequence {Iorg(1), Iorg(2), ..., Iorg(t), ..., Iorg(T)} from the video in frame order, where T is the total number of video frames;

[0040] Step 2: Initialize the first frame of optimal style transfer image

[0041] For the first frame image Iorg(1) in the video, the image style transfer module generates rIax candidate style transfer images Itran(1,ri), where ri is the index of the candidate style transfer image, ranging from 1 to rIax;

[0042] Select the image that minimizes the total loss function of the image style transfer module from the rIax candidate style transfer images Itran(1,ri) as the optimal style transfer image Itran(1,op) of the first frame, or manually select the image with the best user experience from the rIax candidate style transfer images Itran(1,ri) as the optimal style transfer image Itran(1,op) of the first frame, where op represents the optimal.

[0043] Step 3, iteratively generate the optimal style transfer image frame by frame;

[0044] Obtain the optimal style migration image Itran(t,op) of the t+1th frame image based on the optimal style migration image Itran(t,op) of the tth frame image;

[0045] Use the image style transfer module to perform style transfer on the t+1th frame image in the video, and obtain rIax candidate style transfer images Itran(t+1,ri), where the value of ri ranges from 1 to rIax; calculate the inconsistency index between the optimal style transfer image Itran(t,op) of the tth frame image and each candidate style transfer image Itran(t+1,ri);

[0046] The inconsistency index includes a content inconsistency index and a lighting inconsistency index.

[0047] Using the inconsistency indicator as an additional loss term, updating the loss function of the style transfer module;

[0048] Based on the principle of minimizing the updated loss function, the optimal style transfer image Itran(t+1,op) of the t+1th frame image is selected from the rIax candidate style transfer images Itran(t+1,ri).

[0049] Step 4: Determine whether frame t+1 is the last frame in the video. If not, set t = t+1 and repeat step 3 until all frames in the video are processed to generate the style-transferred image sequence {Itran(1,op), Itran (2,op), …, Itran(t,op), …, Itran (T,op)}}.

[0050] In the video generation method for electronic photo wall style transfer consistency of the present invention, a previous and next frame content inconsistency assessment module is used to calculate the content inconsistency index, and a previous and next frame lighting inconsistency assessment module is used to calculate the lighting inconsistency index;

[0051] Module 1. Frame inconsistency assessment module

[0052] The core module of the present invention is the module for assessing the content inconsistency between previous and next frames. Its function is to quickly, roughly, and objectively assess the difference area (DA) and difference level (DB) between the previous and next frame images. In this module, the difference characteristics of the video frame data are analyzed to statistically generate a content inconsistency index. Based on this content inconsistency index, the difference area DA and difference level DB are dynamically constrained during the style transfer image generation stage to ensure that the jump in the content between the previous and next frames does not exceed a preset acceptable range.

[0053] The optimal style transfer image Itran(t,op) of frame t and the ri-th round style transfer image Itran(t+1,ri) of frame t+1 are input into the previous and next frame content inconsistency assessment module for analysis, and the previous and next frame content inconsistency difference Dorg(t) between image Itran(t,op) and image Itran(t+1,ri) is calculated. The previous and next frame content inconsistency difference Dorg(t) includes the difference area DA(t,ri) and the difference level DB(t,ri) between the previous and next frames.

[0054] Normally, the difference between the two images Itran(t,op) and Itran(t+1,ri) after style transfer should be as small as possible. This is the basic principle of the design of the present invention.

[0055] The module for evaluating the inconsistency of the content of the previous and next frames mainly adopts the algorithm Multi-Granularity remote sensing Image Change Detection model (MGCDT) in the paper "Multi-granularity siamese transformer-based change detection in remote sensing imagery" (Volume 136, No. 108960, 2024) in the journal Engineering Applications of Artificial Intelligence. The reason for choosing MGCDT in this paper is that MGCDT can automatically find the common content of the two images, reducing the problem of small-scale non-overlap and mismatch of edges caused by the movement of the user's mobile phone. Only the common parts can be analyzed. The MGCDT structure in the paper includes a Siamese dual-branch encoder, a multi-granularity feature fusion module, a Deep Feature Guidance Unit (DFG), and a multi-scale analysis module.

[0056] To be applicable to the present invention, the present invention makes three improvements to the MGCDT in the paper:

[0057] To reduce computational complexity, this paper tailors the MGCDT architecture, eliminating the computationally intensive multi-scale analysis portion and using only the Stage 1 portion. This allows for rapid and approximate calculation of the Change Map (CM) between two images.

[0058] The front and back frame content inconsistency assessment module of the present invention includes a Siamese dual-branch encoder, a multi-granularity feature fusion module and a deep feature guidance unit (Deep Feature Guidance Unit, DFG);

[0059] The Siamese dual-branch encoder is used to extract the multi-scale features of the input image Itran(t,op) and the multi-scale features of the image Itran(t+1,ri), respectively. In the multi-granularity feature fusion module, a difference feature map is generated based on the multi-scale features of the image Itran(t,op) and the multi-scale features of Itran(t+1,ri), and the difference feature map is input into the Deep Feature Guidance Unit (DFG) to generate a difference map Change Map.

[0060] 2) Because the MGCDT in the paper only outputs a change map between two images, representing only the extent of change (the difference area DA between the previous and next frames), it lacks the specific difference level DB data required by the present invention. Therefore, the present invention extracts the average of the results from the MGCDT's deep feature guidance unit as the difference level DB data for the two images. This allows for a rough and objective assessment of the difference level DB between the two images' change maps.

[0061] 3). Finally, for the Change Map part, count the number of pixels in the Change Map Pcm(t), and multiply the number of pixels Pcm(t) by the difference level DB to obtain the inconsistency index Dd.

[0062] Dd = Pcm*DB (1)

[0063] The inconsistency index of the optimal style transfer image Itran(t,op) of the tth frame and the candidate style transfer image Itran(t+1,ri) of the t+1th frame is expressed as:

[0064] Dd(t,op;t+1,ri)=Pcm(t,op;t+1,ri)*DB(t,op;t+1,ri) (2)

[0065] Then, we get the content inconsistency loss .

[0066] Module 2. Descaling Module (DFM)

[0067] The illumination inconsistency assessment module quickly, roughly, and objectively evaluates the loss function for optical flow changes between frames. Traditional image style transfer doesn't account for the impact of motion (both camera and object motion) on content. This results in the generated image sometimes seeing a single, multiple, or diffuse light source that constantly changes, with the position of the light source fluctuating. This severely impacts the effectiveness of style transfer, so it's important to consider this factor from the very first frame.

[0068] The optical flow method used in the present invention is the classic Lucas-Kanade optical flow method (sparse optical flow method, abbreviated as LK, the original text is Lucas, BD, & Kanade, T. (1981). Title: An iterative image registration technique with an application to stereo vision. International Joint Conference on Artificial Intelligence, Volume 81, No. 1, pages 674-679).

[0069] The steps for calculating the illumination inconsistency index using the previous and next frame illumination inconsistency assessment module are as follows:

[0070] Step 2.1, for the optimal style transfer image Itran(t,op) of the t-th frame image and the candidate style transfer image Itran(t+1,ri) of the t+1 frame, use the difference area DA obtained by the previous and next frame content inconsistency evaluation module (DEM). According to the difference area DA, the difference area images IDAorg(t) and IDAorg(t+1) in the image Itran(t,op) and the image Itran(t+1,ri) are obtained, and the area corresponding to the difference area DA is removed from the image Itran(t,op) and the image Itran(t+1,ri), leaving the indifferent area images InonDAorg(t) and InonDAorg(t+1).

[0071] In step 2.2, the LK optical flow method is used to calculate the optical flows FlowIDAorg(t) and FlowIDAorg(t+1) for the difference area images IDAorg(t) and IDAorg(t+1), respectively.

[0072] Because optical flows are angled and cannot be directly subtracted, the cosine theorem of the vector space model (VSM) is used to calculate the relationship between the optical flows FlowIDAorg(t) and FlowIDAorg(t+1) for each corresponding pixel, resulting in the optical flow change feature matrix SFIDAorg(t,t+1) for the difference region. Larger values ​​in this matrix indicate smaller optical flow changes between corresponding pixels in the previous and next frames; smaller values ​​indicate larger optical flow changes between corresponding pixels in the previous and next frames.

[0073] Then, the average value avgSFIDAorg(t,t+1) and variance sigSFIDAorg(t,t+1) of the optical flow change feature matrix SFIDAorg(t,t+1) are calculated.

[0074] In step 2.3, randomly select 10 points from the indifference region images InonDAorg(t) and InonDAorg(t+1) (this is done to reduce computational complexity and to roughly extract the unchanged portions). Calculate the optical flows FlowInonDAorg(t) and FlowInonDAorg(t+1) using the LK (Lucas-Kanade) optical flow method. Then, also using the vector space model (VSM), calculate the optical flow change feature matrix SFInonDAorg(t, t+1) for these 10 points.

[0075] Then, the average value avgSFInonDAorg(t,t+1) and variance sigSFInonDAorg(t,t+1) of the optical flow change feature matrix SFInonDAorg(t,t+1) are calculated.

[0076] Step 2.4, construct the illumination inconsistency index as the loss of optical flow change :

[0077] (3)

[0078] Module 3. Image Style Transfer Module CGAN

[0079] The backbone network of the image style transfer module of the present invention is CycleGAN (Cycle-Consistent Generative Adversarial Network). CycleGAN is the core contribution of the paper "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks" published by Zhu JY, Park T, Isola P, et al. at the 2017 ICCV conference. It is an innovative framework for achieving unpaired image translation. This means that CycleGAN can learn to translate images from one domain to another without requiring a one-to-one correspondence between the images in the two domains. CycleGAN has had a profound impact on the field of image translation, breaking the limitations of paired data and enabling image translation technology to be applied in a wider range of scenarios. Many subsequent studies have improved and expanded upon the CycleGAN framework, realizing applications in fields such as image style transfer, image enhancement, and data augmentation. Based on this, the present invention modifies some of its structures to make it suitable for use in this scenario.

[0080] The core idea of ​​CycleGAN is to introduce cycle consistency constraints, a method for solving the problem of unpaired image translation. Traditional image translation models rely on paired data, meaning that the training data must contain corresponding images in the source and target domains. However, in many real-world applications, obtaining paired data for the source and target domains is difficult or even impossible. CycleGAN addresses this problem by training two generators and two discriminators.

[0081] like Figure 2 As shown in (a), the first generator G1 converts the original image I(org) from the source domain (original style) Style(org) to the target domain (target style) Style(targ), obtaining its converted image I(tran) in the target domain. The second generator G2 converts the image I(tran) from the target domain Style(targ) back to the source domain Style(org), obtaining the image I(org,inv). The key is that CycleGAN introduces a cycle consistency loss, which requires that for any image I(org) in the source domain, the converted and restored image I(org,inv) should be as close to the original image I(org) as possible.

[0082] Likewise, Figure 2 As shown in (b), for any original image I(targ) in the target domain, the transformed image I(tran) in the source domain is obtained through G2, and then the target domain image I(targ,inv) is obtained through G1. It should be as close to the original image I(targ) as possible. This cycle consistency constraint effectively learns the mapping relationship between the two domains and ensures the rationality of the transformation even without paired data.

[0083] CycleGAN uses an adversarial loss to ensure the authenticity of generated images. Discriminator D1 is used to distinguish between images I(tran) = G1(I(org)) generated by generator G1 and real images in the target domain Style(targ). Similarly, discriminator D2 is used to distinguish between images G2(I(tran)) generated by generator G2 and real images in the source domain Style(org). Through adversarial training, generators G1 and G2 are able to generate realistic images, making it difficult for the discriminator to distinguish between real and fake.

[0084] A key issue here is that CycleGAN's advantage is its use of unpaired data, which reduces its reliance on datasets. However, its disadvantage is also obvious: it is only suitable for processing single-frame images and is not suitable for generating continuous video frames. Therefore, it needs to be modified. The core of the modification lies in CycleGAN's loss function.

[0085] The classic CycleGAN loss functions mainly include two, namely Adversarial Loss and Cycle Consistency Loss.

[0086] Adversarial Loss The core goal is to drive the generator to produce realistic images so that the discriminator cannot distinguish between the generated results and real data.

[0087] Cycle Consistency Loss Ensure that the image can be restored to its original input after bidirectional mapping to prevent the model from learning meaningless mappings.

[0088] In the present invention, Adversarial Loss and Cycle Consistency Loss All calculations are based on the relevant content in the classic paper "Unpaired Image-to-Image translation using cycle-consistent adversarial networks".

[0089] The total expression of the classic loss function is the weighted sum of adversarial loss and cycle consistency loss:

[0090] (4)

[0091] in 、 is a hyperparameter, which can be set by and The value of , adjusts the CycleGAN performance, usually .

[0092] In the present invention, Two loss indicators are added, content inconsistency indicator and illumination inconsistency indicator; the content inconsistency indicator is the result calculated by the previous and next frame inconsistency evaluation module (DEM); the illumination inconsistency indicator is the result calculated by the previous and next frame illumination inconsistency evaluation module (DFM), and the new overall loss function is , which is expressed as follows:

[0093] (5)

[0094] in 、 These are hyperparameters, and users can adjust the overall performance by setting their values. Usually , Indicates content inconsistency loss, Represents the loss of optical flow change.

[0095] Therefore, in the process of initializing the first frame of the optimal style transfer image in step 2 of the present invention, there is no content inconsistency indicator, and the overall loss function of formula (5) is in and Directly equal to 0, the total loss function of the image style transfer module is Equivalent to the classic overall loss function of formula (4) . When the present invention is used in practice, when the first frame of the optimal style transfer image in the video is automatically or manually selected, two frames of optimal style transfer images are automatically generated based on the method of the present invention; then, based on the second frame of the optimal style transfer image, the third frame of the optimal style transfer image is generated; and so on, the optimal style transfer image in the entire video is generated, and the range of its changes before and after is controlled. In this way, by implementing the present invention, users can face the scene of electronic photo walls and effectively solve the two problems of inconsistent content and inconsistent lighting of the previous and next frames encountered in the application of existing classic transfer algorithms in video style conversion, and the requirements for computing power are not large.

Claims

1. A video generation method with style transfer consistency, characterized by: Based on the optimal style transfer image Itran(t,op) of the t-th frame image, the optimal style transfer image Itran(t,op) of the t+1-th frame image is obtained, where t≥1 and op represents the optimal. The specific process is: Use the image style transfer module to perform style transfer on the t+1th frame image in the video, and obtain rIax candidate style transfer images Itran(t+1,ri), where ri is the index of the candidate style transfer image, ranging from 1 to rIax; calculate the inconsistency index between the optimal style transfer image Itran(t,op) of the tth frame image and the candidate style transfer image Itran(t+1,ri); Adding the inconsistency indicator as an inconsistency loss to the loss function of the image style transfer module to obtain an updated loss function; The inconsistency index includes a content inconsistency index, and the calculation method of the content inconsistency index is as follows: Input the optimal style transfer image Itran(t,op) of the t-th frame image and the candidate style transfer image Itran(t+1,ri) into the previous and next frame content inconsistency assessment module, obtain the difference map CM between the image Itran(t,op) and the image Itran(t+1,ri), and count the number of pixels Pcm in the difference map CM; calculating a difference level DB based on the difference map CM; The content inconsistency index Dd is expressed as: Dd(t,op;t+1,ri)=Pcm(t,op;t+1,ri)*DB(t,op;t+1,ri); The inconsistency indicator Dd is used as the content inconsistency loss Added to the loss function of the image style transfer module; The inconsistency index also includes an illumination inconsistency index, and the calculation method of the illumination inconsistency index is as follows: The difference map CM output by the previous and next frame content inconsistency assessment module includes the difference area DA of the previous and next frames. Based on the difference area DA, difference area images IDAorg(t) and IDAorg(t+1) are extracted from the image Itran(t,op) and the image Itran(t+1,ri), respectively, and indifference area images InonDAorg(t) and InonDAorg(t+1) are extracted from the image Itran(t,op) and the image Itran(t+1,ri), respectively. For the difference area images IDAorg(t) and IDAorg(t+1), the optical flow method is used to calculate the optical flows FlowIDAorg(t) and FlowIDAorg(t+1); The cosine theorem formula of the vector space model is used to calculate the relationship between each corresponding pixel of the optical flow FlowIDAorg(t) and FlowIDAorg(t+1), and the optical flow change feature matrix SFIDAorg(t,t+1) of the difference area is obtained; Calculate the average value avgSFIDAorg(t,t+1) and variance sigSFIDAorg(t,t+1) of the optical flow change feature matrix SFIDAorg(t,t+1); Calculate the optical flow change feature matrix SFInonDAorg(t,t+1) of the indifference area images InonDAorg(t) and InonDAorg(t+1), and calculate the average value avgSFInonDAorg(t,t+1) and variance sigSFInonDAorg(t,t+1) of the optical flow change feature matrix SFInonDAorg(t,t+1); Constructing an illumination inconsistency metric as a loss of optical flow variation Added to the loss function of the image style transfer module, the loss of optical flow change It is expressed as follows: ； Based on the principle of minimizing the updated loss function, the optimal style transfer image Itran(t+1,op) of the t+1th frame image is selected from the rIax candidate style transfer images Itran(t+1,ri).

2. The method for generating a video with style transfer consistency according to claim 1, wherein: The calculation of the difference level DB based on the difference map CM is specifically as follows: For each pixel in the difference map CM, the difference value of the feature vector at the corresponding position of the image Itran(t,op) and the image Itran(t+1,ri) is calculated, and the difference values ​​are averaged to obtain the difference level DB.

3. The method for generating a video with style transfer consistency according to claim 1, wherein: The previous and next frame content inconsistency assessment module adopts a pruned MGCDT structure, including a Siamese dual-branch encoder, a multi-granularity feature fusion module, and a deep feature guidance unit; The Siamese dual-branch encoder is used to extract the multi-scale features of the input image Itran(t,op) and the multi-scale features of the image Itran(t+1,ri), respectively, and then input them into the multi-granularity feature fusion module to generate a difference feature map, and the difference feature map is input into the deep feature guidance unit for feature enhancement to generate a difference map CM.

4. The method for generating a video with style transfer consistency according to claim 1, wherein: The method for obtaining the optimal style transfer image Itran(t,op) of the first frame is as follows: The image style transfer module is used to transfer the style of the first frame image in the video, and rIax candidate style transfer images are obtained, so that the loss function of the style transfer module Based on the minimum principle, the optimal style transfer image Itran(1,op) of the first frame image is selected from the rIax candidate style transfer images.

5. The method for generating a video with style transfer consistency according to claim 1, wherein: The inconsistency index is added as the inconsistency loss to the loss function of the image style transfer module to obtain the updated loss function , expressed as ； in, 、 、 、 is a hyperparameter, Represents resistance to loss, represents the cycle consistency loss, Indicates content inconsistency loss, Represents the loss of optical flow change.

6. The method for generating a video with style transfer consistency according to claim 1, wherein: The backbone network of the image style transfer module is CycleGAN.

7. The method for generating a video with style transfer consistency according to claim 1, wherein: When calculating the optical flow change feature matrix SFInonDAorg(t,t+1) of the indifference area images InonDAorg(t) and InonDAorg(t+1), specifically: randomly select no more than 10 points, use the optical flow method to calculate the optical flow FlowInonDAorg(t) and FlowInonDAorg(t+1), and also use the vector space model VSM method to calculate the optical flow change feature matrix SFInonDAorg(t,t+1) of the indifference area of ​​all points.

Images