Real-time video processing method and system in augmented reality environment

Abstract

The invention relates to the technical field of video production, and discloses a real-time video processing method and system in an augmented reality environment, and the method comprises the steps: constructing graph structure data based on a concept image sequence, a split text sequence and a scene text sequence, one data element is associated with one concept image or a split text or a scene text, and the relevance between the two data elements is defined according to the following rules: inputting graph structure data into a trained enhancement agent, and outputting an enhancement result of one concept image of a concept image sequence; according to the method, the scene-divided script and the mirror-divided script are separated, the sparse processing is carried out to correspond to the concept image, and finally, the sparse script semantics and the concept image features are fused through the model, so that local pertinence enhancement of the concept image corresponding to script artistic expression is realized; the enhanced concept image can provide better reference for video production.

Description

Technical Field

[0001] The present invention relates to the technical field of video production, and more specifically, to a real-time video processing method and system in an augmented reality environment. Background Art

[0002] Before starting work on an animated or live-action film, TV series, or video game, the project's production team may hire artists or designers to help define and develop the various characters and scenes included in the project. These hired artists and designers are responsible for the concept development of the project, and are also known as the project's concept artists. Concept artists will draw sketches or illustrations based on the creators, writers, or producers' partial views and imaginations of the project to help the production design team turn the original ideas into realistic graphic images, called concept images; at this stage, concept designers need to have a comprehensive grasp of post-production processes and techniques, and use all existing conditions to provide post-production with visual references that are almost indistinguishable from the final film effect.

[0003] Concept art usually deals with production staff, so video producers need to obtain references and expectations for post-production and actual scene layout from concept images. Therefore, they need to improve the resolution and enhance the details of concept images. Traditional image enhancement algorithms are limited to making images "clearer" and more realistic. However, concept images require a combination of reality and virtuality so that production staff can know what content should be highlighted during production. Existing image enhancement algorithms cannot meet the needs of video production for enhancing concept images. Summary of the invention

[0004] The present invention provides a real-time video processing method in an augmented reality environment, which solves the technical problem that the existing image enhancement algorithm in the related art cannot meet the requirements of video production for enhancing conceptual images.

[0005] The present invention provides a real-time video processing method in an augmented reality environment, comprising the following steps:

[0006] Step 101, sorting the film and television concept images belonging to a film and television drama according to the development order of the described plot to obtain a concept image sequence;

[0007] Step 102, sorting the scene texts of the storyboard script according to the sequence of shots to obtain a storyboard text sequence, where one storyboard text corresponds to the scene text of one shot;

[0008] Sort the scene scripts according to the order of the plot to obtain a scene text sequence, where one scene text corresponds to the entire text of one scene;

[0009] Step 103, constructing graph structure data based on the concept image sequence, the storyboard text sequence and the scene text sequence. The graph structure data includes basic data elements. One data element is associated with one concept image or storyboard text or scene text. The association between two data elements is defined according to the following rules:

[0010] The two data elements are respectively associated with the film and television concept image and the storyboard text. If the film and television concept image refers to the scene text and / or content text of the lens corresponding to the storyboard text during production, the association is defined as 1, otherwise the association is defined as 0;

[0011] The two data elements are respectively associated with the film and television concept image and the scene text. If the film and television concept image refers to the scene text when it is produced, the association is defined as 1, otherwise the association is defined as 0;

[0012] If two data elements are both associated with storyboard texts, and the two storyboard texts contain more than B identical keywords, the association is defined as 1, otherwise the association is defined as 0;

[0013] If two data elements are both associated with scene texts, and the two scene texts contain more than C identical keywords, the association is defined as 1, otherwise the association is defined as 0;

[0014] Step 104, inputting the graph structure data into the trained augmented agent;

[0015] The enhanced agent outputs an enhanced result of a concept image in the concept image sequence at each execution step.

[0016] Furthermore, the value ranges of A, B, and C are all [1, 20].

[0017] Furthermore, keywords are selected from the scene text and the storyboard text, and words that are considered important are manually selected or the top N words in frequency are selected.

[0018] Furthermore, the enhanced agent includes an image encoding layer, a text encoding layer, a graph learning module and a graph generation module, wherein the image encoding layer is used to input a concept image and output image encoding features, and the text encoding layer is used to input storyboard text and scene text respectively, and output storyboard text encoding features and scene text encoding features respectively;

[0019] The graph learning module is used to input feature representations of data elements and data element association data. The feature representations of data elements associated with film and television concept images are image coding features, the feature representations of data elements associated with storyboard texts are storyboard text coding features, and the feature representations of data elements associated with scene texts are scene text coding features.

[0020] The i-th execution step for the augmented agent includes:

[0021] Input the current graph structure data, and output the data encoding features of the data element associated with the i-th concept image;

[0022] Then, the data encoding features of the data element associated with the i-th concept image output by the graph learning module replace the feature representation of the i-th data element;

[0023] The image encoding layer and text encoding layer only run in the first execution step, while the graph learning module runs in each execution step;

[0024] The graph generation module includes a transformation layer and an upsampling layer, wherein the transformation layer divides the data encoding features into three equal parts as three channels, and each channel is then tensored into a matrix form; the upsampling layer inputs the transformed data encoding features, and then performs upsampling to output the enhanced results of the conceptual image associated with the data element corresponding to the data encoding features.

[0025] Furthermore, the graph learning module includes D layers, 6>D>0, and the calculation formula of the lth layer is as follows:

[0026]

[0027] in is the attention weight of data element u to data element v, N(v) represents the set of data elements associated with data element v, Represents the first weight parameter of the lth layer, which is calculated by the following formula:

[0028]

[0029] a is a learnable attention vector, ∥ represents the vector concatenation operation, represents the second weight parameter of the lth layer;

[0030]

[0031] where x v and x u They represent the feature representations of the vth and uth data elements respectively.

[0032] Furthermore, the enhanced result includes resolution increase and / or style transfer relative to the original concept image.

[0033] Furthermore, the image coding layer includes a CNN layer and a tensor layer. The CNN inputs a conceptual image and outputs image features. The image features are expanded and spliced ​​row by row through the tensor layer to obtain image coding features.

[0034] The size of the image feature is a*b*3, 3 represents the number of channels, and the dimension of the image encoding feature is a*b*3;

[0035] The dimension of the data encoding feature is a*b*3.

[0036] Furthermore, the training method of the enhanced agent includes:

[0037] Step 201, initialize the experience pool, randomly initialize the enhanced agent, the second network unit, the third network unit, and the fourth network unit;

[0038] Step 202, resetting graph structure data;

[0039] Step 203: Input graph structure data s t To enhance the agent to obtain the enhanced result a' of the tth concept image t ;

[0040] The data encoding features of the data element associated with the t-th concept image output by the augmented agent replace the feature representation of the t-th data element to update the graph structure data s t+1 ;

[0041] Get image enhancement evaluation r t ; The enhanced evaluation comes from the evaluation of film and television professionals, and the value range of the evaluation score is [-10, 10];

[0042] A score of 0 means that the evaluation of the video concept image after enhancement is consistent with that of the video concept image before enhancement;

[0043] If the enhanced video concept image is better than the video concept image before enhancement, the score is higher, otherwise the score is lower;

[0044] The termination condition done is that all the concept images in the concept image sequence are enhanced;

[0045] Experience t ,a t ,r t ,s t+1 , done is stored in the experience pool. An experience is a four-tuple s t ,a t ,r t ,s t+1 or t ,a t ,r t ,done;

[0046] Step 204, randomly obtain N experiences from the experience pool;

[0047] Input operating status characteristics t To enhance the intelligent agent to obtain the control feature a' t ;

[0048] A' t andt Input the third network unit to get Q Critic (s t ,a' t );

[0049] Input operating status characteristics t+1 Go to the second network unit to get control feature a t+1 ;

[0050]

[0051] Q TargetCritic (s t+1 ,a t+1 ) represents the input running state characteristic s t+1 and control feature a t+1 : the output of the fourth network unit;

[0052] Q Critic (s t ,a t ) represents the input running state characteristic s t+1 and control feature a t+1 : the output of the fourth network unit;

[0053] γ is a hyperparameter with a value range of [0,1];

[0054] Find the loss and use the gradient descent method to update the network;

[0055]

[0056] The enhanced agent is updated through L1, and the third network unit is updated through L2;

[0057] Step 205, every fixed number of network updates, update the second network unit so that its parameters are the same as those of the current augmented agent, and update the fourth network unit so that its parameters are the same as those of the current third network unit;

[0058] Step 206, when the enhanced agent converges and the third network unit converges, the step is terminated or when the number of cycles of the loop steps 203-205 reaches a set value, the step is terminated.

[0059] A real-time video processing system in an augmented reality environment, comprising:

[0060] An image processing module, which sorts the film and television concept images belonging to a film and television drama according to the development order of the described plot to obtain a concept image sequence;

[0061] The text processing module sorts the scene texts of the storyboard script according to the sequence of shots to obtain a storyboard text sequence, where one storyboard text corresponds to the scene text of one shot;

[0062] Sort the scene scripts according to the order of the plot to obtain a scene text sequence, where one scene text corresponds to the entire text of one scene;

[0063] The data processing module constructs graph structure data based on the concept image sequence, the storyboard text sequence and the scene text sequence;

[0064] The image enhancement module inputs the graph structure data into the trained enhanced agent; the enhanced agent outputs the enhanced result of a concept image in the concept image sequence at each execution step.

[0065] A computer storage medium for storing computer-readable instructions, which, when read by a computer, can execute the steps of the method for real-time video processing in an augmented reality environment

[0066] The beneficial effects of the present invention are as follows: the present invention separates the scene-based script and the storyboard script, and performs sparse processing to correspond to the conceptual image, and finally fuses the sparse script semantics and conceptual image features through a model to achieve local targeted enhancement of the conceptual image corresponding to the artistic expression of the script. The enhanced conceptual image can provide a better reference for video production. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a flow chart of a real-time video processing method in an augmented reality environment of the present invention;

[0068] Figure 2 It is a comparison diagram of the film and television concept diagram before and after enhancement of the present invention;

[0069] Figure 3 is the text format of the storyboard of the present invention;

[0070] Figure 4 It is the text format of the scene-by-scene script of the present invention;

[0071] Figure 5 It is a module schematic diagram of a real-time video processing system in an augmented reality environment of the present invention. DETAILED DESCRIPTION

[0072] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the present specification. Various examples may omit, replace, or add various processes or components as needed. In addition, the features described in some examples may also be combined in other examples.

[0073] At least one embodiment of the present invention discloses a real-time video processing method in an augmented reality environment, such as Figure 1 As shown, the following steps are included:

[0074] Step 101, sorting the film and television concept images belonging to a film and television drama according to the development order of the described plot to obtain a concept image sequence;

[0075] Step 102, sorting the scene texts of the storyboard script according to the sequence of shots to obtain a storyboard text sequence, where one storyboard text corresponds to the scene text of one shot;

[0076] Sort the scene scripts according to the order of the plot to obtain a scene text sequence, where one scene text corresponds to the entire text of one scene;

[0077] Combined with the above division Figure 2 , 3 It can be seen from the examples of scene scripts and storyboard scripts that the film and television concept image is only associated with part of the storyboard script or scene script. Therefore, only extracting the script associated with the film and television concept image is an incomplete sequence. Therefore, although the complete storyboard text sequence and scene text sequence are complete, they need to be sparse in order to be well associated with the film and television concept image. The sparseness here does not only mean deleting unrelated sequences, because the sequence units are directly associated;

[0078] For example, the 20th scene in the scene-by-scene script describes the protagonist's initial sword, the 26th scene describes the upgraded sword after the protagonist's initial sword changes, and the 30th storyboard script describes a fight scene including the sword upgrade process. The 30th storyboard script is associated with a film and television concept image that describes the shot. The three texts 20, 26, and 30 are non-continuous and are not even in the same text sequence.

[0079] Therefore, the present invention provides a method for thinning storyboard scripts and scene scripts, aiming to thin the storyboard scripts and scene scripts by combining the image information of the film and television concept image, retaining the key text semantic information without losing the relevance of the plot;

[0080] Since there is a jump mapping relationship between sequence units, the present invention does not consider using a temporal model and an attention mechanism, but instead uses defined rules to construct graph structure data to represent the semantic relationship between sequences, as described in the following steps.

[0081] Step 103, constructing graph structure data based on the concept image sequence, the storyboard text sequence and the scene text sequence. The graph structure data includes basic data elements. One data element is associated with one concept image or storyboard text or scene text. The association between two data elements is defined according to the following rules:

[0082] The two data elements are respectively associated with the film and television concept image and the storyboard text. If the film and television concept image refers to the scene text and / or content text of the lens corresponding to the storyboard text during production, the association is defined as 1, otherwise the association is defined as 0;

[0083] The two data elements are respectively associated with the film and television concept image and the scene text. If the film and television concept image refers to the scene text when it is produced, the association is defined as 1, otherwise the association is defined as 0;

[0084] If two data elements are both associated with storyboard texts, and the two storyboard texts contain more than B identical keywords, the association is defined as 1, otherwise the association is defined as 0;

[0085] If two data elements are both associated with scene texts, and the two scene texts contain more than C identical keywords, the association is defined as 1, otherwise the association is defined as 0;

[0086] Keywords are selected from scene texts and storyboard texts, and are either words that are considered important or the top N words in frequency.

[0087] The larger the values ​​of A, B, and C are, the higher the threshold for capturing the correlation between the context is, which helps to reduce the amount of data processing, but the larger the value is, the better.

[0088] For an example of a TV series suitable for short videos, since the script is more concise, the values ​​of A, B, and C tend to be smaller to capture the less relevant script text. The default values ​​of A, B, and C are 3.

[0089] For movies with more complex scripts, the values ​​of A, B, and C tend to be larger, otherwise the sequence units will be fully connected.

[0090] The value range of A, B, and C is generally between [1, 20].

[0091] Step 104, inputting the graph structure data into the trained augmented agent;

[0092] The enhanced agent outputs an enhanced result of a concept image in the concept image sequence at each execution step;

[0093] The enhanced agent includes an image encoding layer, a text encoding layer, a graph learning module and a graph generation module, wherein the image encoding layer is used to input a concept image and output image encoding features, and the text encoding layer is used to input storyboard text and scene text respectively, and output storyboard text encoding features and scene text encoding features respectively;

[0094] The graph learning module is used to input feature representations of data elements and data element association data. The feature representations of data elements associated with film and television concept images are image coding features, the feature representations of data elements associated with storyboard texts are storyboard text coding features, and the feature representations of data elements associated with scene texts are scene text coding features.

[0095] The i-th (N>i>0, N represents the total number of concept images in the concept image sequence) execution step of the enhanced agent includes:

[0096] Input the current graph structure data, and output the data encoding features of the data element associated with the i-th concept image;

[0097] Then, the data encoding features of the data element associated with the i-th concept image output by the graph learning module replace the feature representation of the i-th data element;

[0098] The image encoding layer and text encoding layer only run in the first execution step, while the graph learning module runs in each execution step;

[0099] The graph generation module includes a transformation layer and an upsampling layer, wherein the transformation layer divides the data encoding features into three equal parts as three channels, and each channel is then tensored into a matrix form; the upsampling layer inputs the transformed data encoding features, and then performs upsampling to output the enhanced results of the conceptual image associated with the data element corresponding to the data encoding features.

[0100] In one embodiment of the present invention, the graph learning module includes D layers, 6>D>0, and the calculation formula of the lth layer is as follows:

[0101]

[0102] in is the attention weight of data element u to data element v, N(v) represents the set of data elements associated with data element v, Represents the first weight parameter of the lth layer, which is calculated by the following formula:

[0103]

[0104] a is a learnable attention vector, ∥ represents the vector concatenation operation, represents the second weight parameter of the lth layer;

[0105]

[0106] where x v and x u Respectively represent the feature representation of the vth and uth data elements;

[0107] In one embodiment of the present invention, the enhancement result includes resolution increase and / or style transfer relative to the original concept image.

[0108] In one embodiment of the present invention, the image coding layer includes a CNN layer and a tensor layer, the CNN inputs a conceptual image and outputs image features, and the image features are expanded and spliced ​​row by row through the tensor layer to obtain image coding features;

[0109] The size of the image feature is a*b*3, 3 represents the number of channels, generally corresponding to the three RGB channels, and the dimension of the image encoding feature is a*b*3;

[0110] The dimension of the data encoding feature is a*b*3.

[0111] In one embodiment of the present invention, the text encoding layer uses a one-hot encoding method to perform text encoding.

[0112] In one embodiment of the present invention, the text encoding layer is encoded using a text encoding model such as word2c.

[0113] In one embodiment of the present invention, the training method of the enhanced agent includes:

[0114] Step 201, initialize the experience pool, randomly initialize the enhanced agent, the second network unit, the third network unit, and the fourth network unit;

[0115] Step 202, resetting graph structure data;

[0116] Step 203: Input graph structure data s t To enhance the agent to obtain the enhanced result a' of the tth concept image t ;

[0117] The data encoding features of the data element associated with the t-th concept image output by the augmented agent replace the feature representation of the t-th data element to update the graph structure data s t+1 ;

[0118] Get image enhancement evaluation r t ; The enhanced evaluation comes from the evaluation of film and television professionals, and the value range of the evaluation score is [-10, 10];

[0119] A score of 0 means that the evaluation of the video concept image after enhancement is consistent with that of the video concept image before enhancement;

[0120] If the enhanced video concept image is better than the video concept image before enhancement, the score is higher, otherwise the score is lower;

[0121] The termination condition done is that all the concept images in the concept image sequence are enhanced;

[0122] Experience t ,a t ,r t ,s t+1 , done is stored in the experience pool. An experience is a four-tuple s t ,a t ,r t ,s t+1 or t ,a t ,r t ,done;

[0123] Step 204, randomly obtain N experiences from the experience pool;

[0124] Input operating status characteristics t To enhance the intelligent agent to obtain the control feature a' t ;

[0125] A' t and t Input the third network unit to get Q Critic (s t ,a' t );

[0126] Input operating status characteristics t+1 Go to the second network unit to get control feature a t+1 ;

[0127]

[0128] Q TargetCritic (s t+1 ,a t+1 ) represents the input running state characteristic s t+1 and control feature a t+1 : the output of the fourth network unit;

[0129] Q Critic (s t ,a t ) represents the input running state characteristic s t+1 and control feature a t+1 : the output of the fourth network unit;

[0130] γ is a hyperparameter, the range is [0,1], and the default value is 0.6;

[0131] Find the loss and use the gradient descent method to update the network;

[0132]

[0133] The enhanced agent is updated through L1, and the third network unit is updated through L2;

[0134] Step 205, every fixed number of network updates, update the second network unit so that its parameters are the same as those of the current augmented agent, and update the fourth network unit so that its parameters are the same as those of the current third network unit;

[0135] Step 206, when the enhanced agent converges and the third network unit converges, the step is terminated or when the number of cycles of the loop steps 203-205 reaches a set value, the step is terminated.

[0136] In one embodiment of the present invention, the training method of the enhanced intelligent agent is to perform supervised training based on the results of artificial hand-drawing (including software hand-drawing) enhancement as supervision, and the loss function is a cross entropy loss function.

[0137] The third network unit and the fourth network unit can both adopt MLP or CNN or GNN or RNN.

[0138] In at least one embodiment of the present invention, a real-time video processing system in an augmented reality environment is provided. Figure 5 As shown, including:

[0139] The image processing module 301 arranges the film and television concept images belonging to a film and television drama in the order of the development of the described plot to obtain a concept image sequence;

[0140] The text processing module 302 sorts the scene texts of the storyboard script according to the sequence of shots to obtain a storyboard text sequence, where one storyboard text corresponds to the scene text of one shot;

[0141] Sort the scene scripts according to the order of the plot to obtain a scene text sequence, where one scene text corresponds to the entire text of one scene;

[0142] A data processing module 303 constructs graph structure data based on the concept image sequence, the storyboard text sequence and the scene text sequence;

[0143] The image enhancement module 304 inputs the graph structure data into the trained enhanced agent; the enhanced agent outputs the enhancement result of a concept image in the concept image sequence at each execution step.

[0144] In at least one embodiment of the present invention, a computer storage medium is provided for storing computer-readable instructions, which can execute the steps of the aforementioned method for real-time video processing in an augmented reality environment when the computer-readable instructions are read by a computer.

[0145] The above describes an embodiment of the present invention, but this embodiment is not limited to the above-mentioned specific implementation mode. The above-mentioned specific implementation mode is merely illustrative and not restrictive. Under the guidance of this embodiment, ordinary technicians in this field can also make more forms of equivalent embodiments, all of which are protected by this embodiment.

[0146] It should be noted that:

[0147] The production of film and television concept drawings includes but is not limited to hand-drawing and computer production;

[0148] The present invention is not applicable to the case where the film and television concept images in a video are produced in different styles. For example, some film and television concept images are in comic style, while others are in realistic style. After such film and television concept image sequences are associated, it is difficult for the model to learn patterns.

[0149] The film and television concept map of the present invention does not include a concept map of an independent individual or object in a local scene.

Claims

1. A real-time video processing method in an augmented reality environment, characterized in that: The following steps are involved: Step 101, sorting the film and television concept images belonging to a film and television drama according to the development order of the described plot to obtain a concept image sequence; Step 102, sorting the scene texts of the storyboard script according to the sequence of shots to obtain a storyboard text sequence, where one storyboard text corresponds to the scene text of one shot; Sort the scene scripts according to the order of the plot to obtain a scene text sequence, where one scene text corresponds to the entire text of one scene; Step 103, constructing graph structure data based on the concept image sequence, the storyboard text sequence and the scene text sequence. The graph structure data includes basic data elements. One data element is associated with one concept image or storyboard text or scene text. The association between two data elements is defined according to the following rules: The two data elements are respectively associated with the film and television concept image and the storyboard text. If the film and television concept image refers to the scene text and / or content text of the lens corresponding to the storyboard text during production, the association is defined as 1, otherwise the association is defined as 0; The two data elements are respectively associated with the film and television concept image and the scene text. If the film and television concept image refers to the scene text when it is produced, the association is defined as 1, otherwise the association is defined as 0; If two data elements are both associated with storyboard texts, and the two storyboard texts contain more than B identical keywords, the association is defined as 1, otherwise the association is defined as 0; If two data elements are both associated with scene texts, and the two scene texts contain more than C identical keywords, the association is defined as 1, otherwise the association is defined as 0; Step 104, inputting the graph structure data into the trained augmented agent; The enhanced agent outputs an enhanced result of a concept image in the concept image sequence at each execution step.

2. The real-time video processing method in an augmented reality environment according to claim 1, characterized in that: The value ranges of A, B, and C are [ 1,20 ] .

3. The real-time video processing method in an augmented reality environment according to claim 1, characterized in that: Keywords are selected from scene texts and storyboard texts, and are either words that are considered important or the top N words in frequency.

4. The real-time video processing method in an augmented reality environment according to claim 1, characterized in that: The enhanced agent includes an image encoding layer, a text encoding layer, a graph learning module and a graph generation module, wherein the image encoding layer is used to input a concept image and output image encoding features, and the text encoding layer is used to input storyboard text and scene text respectively, and output storyboard text encoding features and scene text encoding features respectively; The graph learning module is used to input feature representations of data elements and data element association data. The feature representations of data elements associated with film and television concept images are image coding features, the feature representations of data elements associated with storyboard texts are storyboard text coding features, and the feature representations of data elements associated with scene texts are scene text coding features. The i-th execution step for the augmented agent includes: Input the current graph structure data, and output the data encoding features of the data element associated with the i-th concept image; Then, the data encoding features of the data element associated with the i-th concept image output by the graph learning module replace the feature representation of the i-th data element; The image encoding layer and text encoding layer only run in the first execution step, while the graph learning module runs in each execution step; The graph generation module includes a transformation layer and an upsampling layer, wherein the transformation layer divides the data encoding features into three equal parts as three channels, and each channel is then tensored into a matrix form; the upsampling layer inputs the transformed data encoding features, and then performs upsampling to output the enhanced results of the conceptual image associated with the data element corresponding to the data encoding features.

5. The real-time video processing method in an augmented reality environment according to claim 4, characterized in that: The graph learning module includes D layers, 6>D>0, and the calculation formula of the lth layer is as follows: in is the attention weight of data element u to data element v, N ( v ) represents the set of data elements associated with data element v, Represents the first weight parameter of the lth layer, which is calculated by the following formula: a is a learnable attention vector, ∥ represents the vector concatenation operation, represents the second weight parameter of the lth layer; where x v and x u They represent the feature representations of the vth and uth data elements respectively.

6. The real-time video processing method in an augmented reality environment according to claim 4, characterized in that: The enhanced result may include resolution increase and / or style transfer relative to the original concept image.

7. The real-time video processing method in an augmented reality environment according to claim 4, characterized in that: The image coding layer includes a CNN layer and a tensor layer. The CNN inputs a conceptual image and outputs image features. The image features are expanded and concatenated row by row through the tensor layer to obtain image coding features. The size of the image feature is a*b*3, 3 represents the number of channels, and the dimension of the image encoding feature is a*b*3; The dimension of the data encoding feature is a*b*3.

8. The real-time video processing method in an augmented reality environment according to claim 1, characterized in that: Methods for training augmented agents include: Step 201, initialize the experience pool, randomly initialize the enhanced agent, the second network unit, the third network unit, and the fourth network unit; Step 202, resetting graph structure data; Step 203: Input graph structure data s t To enhance the agent to obtain the enhanced result a of the tth concept image t ， ; The data encoding features of the data element associated with the t-th concept image output by the augmented agent replace the feature representation of the t-th data element to update the graph structure data s t+1 ; Get image enhancement evaluation r t ; The enhanced evaluation comes from the evaluation of film and television professionals, and the value range of the evaluation score is [ -10, 10 ] ; A score of 0 means that the evaluation of the video concept image after enhancement is consistent with that of the video concept image before enhancement; If the enhanced video concept image is better than the video concept image before enhancement, the score is higher, otherwise the score is lower; The termination condition done is that all the concept images in the concept image sequence are enhanced; Experience t ,a t ,r t ,s t+1 , done is stored in the experience pool. An experience is a four-tuple s t ,a t ,r t ,s t+1 or t ,a t ,r t ,done; Step 204, randomly obtain N experiences from the experience pool; Input operating status characteristics t To enhance the intelligent agent to obtain control characteristics a t ， ; A t ， and t Input the third network unit to get Q Critic ( s t ,a t ，) ; Input operating status characteristics t+1 Go to the second network unit to get control feature a t+1 ; Q TargetCritic ( s t+1 ,a t+1 ) Indicates the input running state characteristics s t+1 and control feature a t+1 : the output of the fourth network unit; Q Critic( s t ,a t) Indicates the input running state characteristics s t+1 and control feature a t+1 : the output of the fourth network unit; γ is a hyperparameter with a range of [ 0,1 ] ; Find the loss and use the gradient descent method to update the network; The enhanced agent is updated through L1, and the third network unit is updated through L2; Step 205, every fixed number of network updates, update the second network unit so that its parameters are the same as those of the current augmented agent, and update the fourth network unit so that its parameters are the same as those of the current third network unit; Step 206, when the enhanced agent converges and the third network unit converges, the step is terminated or when the number of cycles of the loop steps 203-205 reaches a set value, the step is terminated.

9. A computer storage medium, characterized in that: It is used to store computer-readable instructions, which, when read by a computer, can execute the steps of the method for real-time video processing in an augmented reality environment as described in any one of claims 1-8.

10. A real-time video processing system in an augmented reality environment, characterized in that: include: An image processing module, which sorts the film and television concept images belonging to a film and television drama according to the development order of the described plot to obtain a concept image sequence; The text processing module sorts the scene texts of the storyboard script according to the sequence of shots to obtain a storyboard text sequence, where one storyboard text corresponds to the scene text of one shot; Sort the scene scripts according to the order of the plot to obtain a scene text sequence, where one scene text corresponds to the entire text of one scene; The data processing module constructs graph structure data based on the concept image sequence, the storyboard text sequence and the scene text sequence; Image enhancement module, which inputs graph structure data into the trained enhanced agent; The enhanced agent outputs an enhanced result of a concept image in the concept image sequence at each execution step.

Images