A real-time video processing method and system in an augmented reality environment

Abstract

The present invention relates to the technical field of video production, and discloses a real-time video processing method and system in an augmented reality environment. The method comprises: constructing graph structure data based on a concept image sequence, a storyboard text sequence, and a scene text sequence, wherein the graph structure data comprises basic data elements, wherein one data element is associated with one concept image, one storyboard text, or one scene text, and the correlation between two data elements is defined according to the following rules: inputting the graph structure data into a trained augmented intelligent agent, and outputting an enhancement result of a concept image in the concept image sequence; the present invention separates the scene script and the storyboard script, and performs sparse processing to correspond to the concept image; finally, the sparse script semantics and the concept image features are integrated into a model to achieve local targeted enhancement of the concept image corresponding to the script artistic expression, and the enhanced concept image can provide a better reference for video production.

Description

Technical Field

[0001] The present invention relates to the field of video production technology, and more particularly, to a real-time video processing method and system in an augmented reality environment. Background Art

[0002] Before beginning production on an animated or live-action film, television series, or video game, the project's production team may hire artists or designers to help define and develop the various characters and scenes included in the project. These hired artists and designers are responsible for developing the project's concepts, and are known as the project's concept artists. Based on the creators, writers, or producers' partial visions and imaginations for the project, concept artists will draw sketches or illustrations to help the production design team transform their original ideas into realistic graphic images, known as concept images. At this stage, concept designers need to have a comprehensive grasp of post-production processes and techniques, utilizing all available resources to provide visual references for post-production that are virtually indistinguishable from the final film.

[0003] Concept art usually faces production staff, so video producers need to obtain references and expectations for post-production and actual scene layout from concept images. Therefore, they need to improve the resolution and enhance the details of concept images. Traditional image enhancement algorithms are limited to making images "clearer" and more realistic. However, concept images require a combination of virtual and real so that producers can know what content should be highlighted during production. Existing image enhancement algorithms cannot meet the needs of video production for enhancing concept images. Summary of the Invention

[0004] The present invention provides a real-time video processing method in an augmented reality environment, which solves the technical problem that the existing image enhancement algorithms in the related art cannot meet the requirements of video production for enhancing conceptual images.

[0005] The present invention provides a real-time video processing method in an augmented reality environment, comprising the following steps:

[0006] Step 101: sort the film and television concept images belonging to a film and television drama according to the development order of the described plot to obtain a concept image sequence;

[0007] Step 102: sorting the scene texts of the storyboard script according to the sequence of shots to obtain a storyboard text sequence, where one storyboard text corresponds to the scene text of one shot;

[0008] Sort the scene scripts in the order of the plot to obtain a scene text sequence, where one scene text corresponds to the entire text of a scene;

[0009] Step 103: Graph structure data is constructed based on the concept image sequence, the storyboard text sequence, and the scene text sequence. The graph structure data includes basic data elements. One data element is associated with one concept image, storyboard text, or scene text. The association between two data elements is defined according to the following rules:

[0010] The two data elements are associated with a film and television concept image and a storyboard text respectively. If the film and television concept image refers to the scene text and / or content text of the shot corresponding to the storyboard text during production, the association is defined as 1; otherwise, the association is defined as 0.

[0011] The two data elements are associated with the film and television concept image and the scene text respectively. If the film and television concept image refers to the scene text when it is produced, the association is defined as 1, otherwise the association is defined as 0;

[0012] If two data elements are both associated with storyboard texts, and the two storyboard texts contain more than B identical keywords, then the correlation is defined as 1, otherwise the correlation is defined as 0;

[0013] If two data elements are both associated with scene texts and the two scene texts contain more than C identical keywords, the correlation is defined as 1, otherwise the correlation is defined as 0;

[0014] Step 104: input the graph structure data into the trained augmented agent;

[0015] The enhanced agent outputs an enhanced result of a concept image in the concept image sequence at each execution step.

[0016] Furthermore, the value ranges of A, B, and C are all [1, 20].

[0017] Furthermore, keywords are selected from the scene text and storyboard text, and words that are considered important are manually selected or the top N words in frequency are selected.

[0018] Furthermore, the enhanced agent includes an image encoding layer, a text encoding layer, a graph learning module, and a graph generation module, wherein the image encoding layer is used to input a concept image and output image encoding features, and the text encoding layer is used to input storyboard text and scene text respectively, and output storyboard text encoding features and scene text encoding features respectively;

[0019] The graph learning module is used to input the feature representation of data elements and the association data of the data elements. The feature representation of data elements associated with film and television concept images is image coding features, the feature representation of data elements associated with storyboard text is storyboard text coding features, and the feature representation of data elements associated with scene text is scene text coding features.

[0020] The i-th execution step for the augmented agent includes:

[0021] Input the current graph structure data and output the data encoding features of the data element associated with the i-th concept image;

[0022] Then, the data encoding features of the data element associated with the i-th concept image output by the graph learning module replace the feature representation of the i-th data element;

[0023] The image encoding layer and text encoding layer only run in the first execution step, while the graph learning module runs in every execution step;

[0024] The graph generation module includes a transformation layer and an upsampling layer. The transformation layer divides the data encoding features into three equal parts as three channels, and each channel is then tensored into a matrix form. The upsampling layer inputs the transformed data encoding features, and then performs upsampling to output the enhanced results of the conceptual image associated with the data element corresponding to the data encoding features.

[0025] Furthermore, the graph learning module includes D layers, 6>D>0, and the calculation formula of the lth layer is as follows:

[0026]

[0027] in is the attention weight of data element u to data element v, N(v) represents the set of data elements associated with data element v, Represents the first weight parameter of the lth layer, which is calculated by the following formula:

[0028]

[0029] a is a learnable attention vector, ∥ represents the vector concatenation operation, represents the second weight parameter of the lth layer;

[0030]

[0031] where x v and x u Represent the feature representation of the vth and uth data elements respectively.

[0032] Furthermore, the enhanced results include resolution increase and / or style transfer relative to the original concept image.

[0033] Furthermore, the image coding layer includes a CNN layer and a tensor layer. The CNN inputs the concept image and outputs image features. The image features are expanded and spliced ​​row by row through the tensor layer to obtain image coding features.

[0034] The size of the image feature is a*b*3, 3 represents the number of channels, and the dimension of the image encoding feature is a*b*3;

[0035] The dimension of the data encoding feature is a*b*3.

[0036] Furthermore, the training method of the enhanced agent includes:

[0037] Step 201: Initialize the experience pool, and randomly initialize the enhanced agent, the second network unit, the third network unit, and the fourth network unit;

[0038] Step 202, resetting the graph structure data;

[0039] Step 203: Input graph structure data s t To enhance the agent to obtain the enhanced result a' of the t-th concept image t ;

[0040] The data encoding features of the data element associated with the t-th concept image output by the augmented agent replace the feature representation of the t-th data element to update the graph structure data s t+1 ;

[0041] Get image enhancement evaluation r t ; The enhanced evaluation comes from the evaluation of film and television professionals, and the evaluation score range is [-10, 10];

[0042] A score of 0 indicates that the evaluation of the video concept image after enhancement is consistent with that before enhancement;

[0043] If the enhanced video concept image is better than the video concept image before enhancement, the score is higher, otherwise the score is lower;

[0044] The termination condition done is that all concept images in the concept image sequence are enhanced;

[0045] The experience t ,a t ,r t ,s t+1 , done is stored in the experience pool, an experience is a four-tuple s t ,a t ,r t ,s t+1 or s t ,a t ,r t ,done;

[0046] Step 204, randomly obtain N experiences from the experience pool;

[0047] Input operating status characteristics s t To enhance the intelligent agent to obtain the control feature a' t ;

[0048] will a' t and st Input the third network unit to get Q Critic (s t ,a' t );

[0049] Input operating status characteristics s t+1 Go to the second network unit to get control feature a t+1 ;

[0050]

[0051] Q TargetCritic (s t+1 ,a t+1 ) represents the input running state characteristics s t+1 and control feature a t+1 The output of the fourth network unit;

[0052] Q Critic (s t ,a t ) represents the input running state characteristics s t+1 and control feature a t+1 The output of the fourth network unit;

[0053] γ is a hyperparameter with a value range of [0,1];

[0054] Find the loss and use the gradient descent method to update the network;

[0055]

[0056] Update the enhanced agent via L1 and update the third network unit via L2;

[0057] Step 205: every fixed number of network updates, update the second network unit so that its parameters are the same as those of the current augmented agent, and update the fourth network unit so that its parameters are the same as those of the current third network unit;

[0058] Step 206, when the enhanced agent converges and the third network unit converges, the step is terminated or when the number of cycles of steps 203-205 reaches a set value, the step is terminated.

[0059] A real-time video processing system in an augmented reality environment, comprising:

[0060] An image processing module, which sorts the film and television concept images belonging to a film and television drama according to the development order of the described plot to obtain a concept image sequence;

[0061] The text processing module sorts the scene texts of the storyboard script according to the sequence of shots to obtain a storyboard text sequence, where one storyboard text corresponds to the scene text of one shot;

[0062] Sort the scene scripts in the order of the plot to obtain a scene text sequence, where one scene text corresponds to the entire text of a scene;

[0063] The data processing module constructs graph structure data based on the concept image sequence, storyboard text sequence, and scene text sequence;

[0064] The image enhancement module inputs the graph structure data into the trained enhanced intelligent agent; the enhanced intelligent agent outputs the enhanced result of a concept image in the concept image sequence at each execution step.

[0065] A computer storage medium for storing computer-readable instructions, which, when read by a computer, can execute the steps of the method for real-time video processing in an augmented reality environment

[0066] The beneficial effects of the present invention are as follows: the present invention separates the scene-by-scene script and the storyboard script, and performs sparse processing to correspond to the conceptual image. Finally, the sparse script semantics and conceptual image features are integrated through the model to achieve local targeted enhancement of the conceptual image corresponding to the artistic expression of the script. The enhanced conceptual image can provide a better reference for video production. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a flow chart of a real-time video processing method in an augmented reality environment of the present invention;

[0068] Figure 2 It is a comparison diagram of the film and television concept map before and after enhancement of the present invention;

[0069] Figure 3 is the text format of the storyboard of the present invention;

[0070] Figure 4 It is the text format of the scene-by-scene script of the present invention;

[0071] Figure 5 This is a module schematic diagram of a real-time video processing system in an augmented reality environment of the present invention. DETAILED DESCRIPTION

[0072] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that these embodiments are discussed solely to enable those skilled in the art to better understand and implement the subject matter described herein, and that the functions and arrangements of the elements discussed may be varied without departing from the scope of this specification. Various examples may omit, substitute, or add various processes or components as needed. Furthermore, features described in some examples may be combined in other examples.

[0073] At least one embodiment of the present invention discloses a real-time video processing method in an augmented reality environment, such as Figure 1 As shown, the following steps are included:

[0074] Step 101: sort the film and television concept images belonging to a film and television drama according to the development order of the described plot to obtain a concept image sequence;

[0075] Step 102: sorting the scene texts of the storyboard script according to the sequence of shots to obtain a storyboard text sequence, where one storyboard text corresponds to the scene text of one shot;

[0076] Sort the scene scripts in the order of the plot to obtain a scene text sequence, where one scene text corresponds to the entire text of a scene;

[0077] From the above division combined with the Figure 2 、 3 From the examples of scene-by-scene scripts and storyboard scripts, we can see that the film and television concept image is only associated with part of the storyboard script or scene-by-scene script. Therefore, only extracting the script associated with the film and television concept image is an incomplete sequence. Therefore, although the complete storyboard text sequence and scene text sequence are complete, they need to be sparsed in order to be well associated with the film and television concept image. The sparseness here does not just mean deleting unrelated sequences, because the sequence units are directly associated.

[0078] For example, the 20th scene in the scene-by-scene script describes the protagonist's initial sword, the 26th scene describes the protagonist's upgraded sword after the initial sword changes, and the 30th storyboard script describes a fight scene including the sword upgrade process. The 30th storyboard script is associated with a film and television concept image describing the shot. The three texts 20, 26, and 30 are non-continuous and are not even in the same text sequence.

[0079] Therefore, the present invention provides a method for thinning storyboards and scene-by-scene scripts, aiming to thin out the storyboards and scene-by-scene scripts by combining the image information of film and television concept images, thereby retaining key text semantic information while maintaining the relevance of the plot.

[0080] Since there is a jump mapping relationship between sequence units, the present invention does not consider using a temporal model and an attention mechanism, but instead uses defined rules to construct graph structure data to represent the semantic relationship between sequences, as described in the following steps.

[0081] Step 103: Graph structure data is constructed based on the concept image sequence, the storyboard text sequence, and the scene text sequence. The graph structure data includes basic data elements. One data element is associated with one concept image, storyboard text, or scene text. The association between two data elements is defined according to the following rules:

[0082] The two data elements are associated with a film and television concept image and a storyboard text respectively. If the film and television concept image refers to the scene text and / or content text of the shot corresponding to the storyboard text during production, the association is defined as 1; otherwise, the association is defined as 0.

[0083] The two data elements are associated with the film and television concept image and the scene text respectively. If the film and television concept image refers to the scene text when it is produced, the association is defined as 1, otherwise the association is defined as 0;

[0084] If two data elements are both associated with storyboard texts, and the two storyboard texts contain more than B identical keywords, then the correlation is defined as 1, otherwise the correlation is defined as 0;

[0085] If two data elements are both associated with scene texts and the two scene texts contain more than C identical keywords, the correlation is defined as 1, otherwise the correlation is defined as 0;

[0086] Keywords are selected from scene texts and storyboard texts, and are manually selected as words that are considered important or the top N words in frequency.

[0087] The larger the values ​​of A, B, and C are, the higher the threshold for capturing the correlation between the context, which helps to reduce the amount of data processing, but the larger the value is, the better.

[0088] For a short-form TV series, the script is relatively concise, so the values ​​of A, B, and C are smaller to capture the less relevant script text. The default values ​​of A, B, and C are 3.

[0089] For movies with more complex scripts, the values ​​of A, B, and C tend to be larger, otherwise the sequence units will be fully connected.

[0090] The value range of A, B, and C is generally between [1, 20].

[0091] Step 104: input the graph structure data into the trained augmented agent;

[0092] The enhanced agent outputs an enhanced result of a concept image in the concept image sequence at each execution step;

[0093] The enhanced agent includes an image encoding layer, a text encoding layer, a graph learning module, and a graph generation module. The image encoding layer is used to input a concept image and output image encoding features. The text encoding layer is used to input storyboard text and scene text respectively, and output storyboard text encoding features and scene text encoding features respectively.

[0094] The graph learning module is used to input the feature representation of data elements and the association data of the data elements. The feature representation of data elements associated with film and television concept images is image coding features, the feature representation of data elements associated with storyboard text is storyboard text coding features, and the feature representation of data elements associated with scene text is scene text coding features.

[0095] The i-th (N>i>0, N represents the total number of concept images in the concept image sequence) execution step of the augmented agent includes:

[0096] Input the current graph structure data and output the data encoding features of the data element associated with the i-th concept image;

[0097] Then, the data encoding features of the data element associated with the i-th concept image output by the graph learning module replace the feature representation of the i-th data element;

[0098] The image encoding layer and text encoding layer only run in the first execution step, while the graph learning module runs in every execution step;

[0099] The graph generation module includes a transformation layer and an upsampling layer. The transformation layer divides the data encoding features into three equal parts as three channels, and each channel is then tensored into a matrix form. The upsampling layer inputs the transformed data encoding features, and then performs upsampling to output the enhanced results of the conceptual image associated with the data element corresponding to the data encoding features.

[0100] In one embodiment of the present invention, the graph learning module includes D layers, 6>D>0, and the calculation formula of the first layer is as follows:

[0101]

[0102] in is the attention weight of data element u to data element v, N(v) represents the set of data elements associated with data element v, Represents the first weight parameter of the lth layer, which is calculated by the following formula:

[0103]

[0104] a is a learnable attention vector, ∥ represents the vector concatenation operation, represents the second weight parameter of the lth layer;

[0105]

[0106] where x v and x u Represent the feature representation of the vth and uth data elements respectively;

[0107] In one embodiment of the present invention, the enhancement result includes resolution increase and / or style transfer relative to the original concept image.

[0108] In one embodiment of the present invention, the image coding layer includes a CNN layer and a tensor layer. The CNN inputs a conceptual image and outputs image features. The image features are expanded and spliced ​​row by row through the tensor layer to obtain image coding features.

[0109] The size of the image feature is a*b*3, where 3 represents the number of channels, generally corresponding to the RGB three channels. The dimension of the image encoding feature is a*b*3;

[0110] The dimension of the data encoding feature is a*b*3.

[0111] In one embodiment of the present invention, the text encoding layer uses a one-hot encoding method to perform text encoding.

[0112] In one embodiment of the present invention, the text encoding layer is encoded using a text encoding model such as word2c.

[0113] In one embodiment of the present invention, the training method of an augmented agent includes:

[0114] Step 201: Initialize the experience pool, and randomly initialize the enhanced agent, the second network unit, the third network unit, and the fourth network unit;

[0115] Step 202, resetting the graph structure data;

[0116] Step 203: Input graph structure data s t To enhance the agent to obtain the enhanced result a' of the t-th concept image t ;

[0117] The data encoding features of the data element associated with the t-th concept image output by the augmented agent replace the feature representation of the t-th data element to update the graph structure data s t+1 ;

[0118] Get image enhancement evaluation r t ; The enhanced evaluation comes from the evaluation of film and television professionals, and the evaluation score range is [-10, 10];

[0119] A score of 0 indicates that the evaluation of the video concept image after enhancement is consistent with that before enhancement;

[0120] If the enhanced video concept image is better than the video concept image before enhancement, the score is higher, otherwise the score is lower;

[0121] The termination condition done is that all concept images in the concept image sequence are enhanced;

[0122] The experience t ,a t ,r t ,s t+1 , done is stored in the experience pool, an experience is a four-tuple s t ,a t ,r t ,s t+1 or s t ,a t ,r t ,done;

[0123] Step 204, randomly obtain N experiences from the experience pool;

[0124] Input operating status characteristics s t To enhance the intelligent agent to obtain the control feature a' t ;

[0125] Will a' t and s t Input the third network unit to get Q Critic (s t ,a' t );

[0126] Input operating status characteristics s t+1 Go to the second network unit to get control feature a t+1 ;

[0127]

[0128] Q TargetCritic (s t+1 ,a t+1 ) represents the input running state characteristics s t+1 and control feature a t+1 The output of the fourth network unit;

[0129] Q Critic (s t ,a t ) represents the input running state characteristics s t+1 and control feature a t+1 The output of the fourth network unit;

[0130] γ is a hyperparameter with a range of [0,1] and a default value of 0.6;

[0131] Find the loss and use the gradient descent method to update the network;

[0132]

[0133] Update the enhanced agent via L1 and update the third network unit via L2;

[0134] Step 205: every fixed number of network updates, update the second network unit so that its parameters are the same as those of the current augmented agent, and update the fourth network unit so that its parameters are the same as those of the current third network unit;

[0135] Step 206, when the enhanced agent converges and the third network unit converges, the step is terminated or when the number of cycles of steps 203-205 reaches a set value, the step is terminated.

[0136] In one embodiment of the present invention, the training method of the enhanced intelligent agent is to perform supervised training based on the enhanced results of manual hand-drawing (including software hand-drawing), and the loss function is a cross-entropy loss function.

[0137] The third network unit and the fourth network unit can both adopt MLP or CNN or GNN or RNN.

[0138] In at least one embodiment of the present invention, a real-time video processing system in an augmented reality environment is provided. Figure 5 Shown, including:

[0139] The image processing module 301 sorts the film and television concept images belonging to a film and television drama according to the development order of the described plot to obtain a concept image sequence;

[0140] The text processing module 302 sorts the scene texts of the storyboard script according to the sequence of shots to obtain a storyboard text sequence, where one storyboard text corresponds to the scene text of one shot;

[0141] Sort the scene scripts in the order of the plot to obtain a scene text sequence, where one scene text corresponds to the entire text of a scene;

[0142] The data processing module 303 constructs graph structure data based on the concept image sequence, the storyboard text sequence, and the scene text sequence;

[0143] The image enhancement module 304 inputs the graph structure data into the trained enhancement agent; the enhancement agent outputs the enhancement result of a concept image in the concept image sequence at each execution step.

[0144] In at least one embodiment of the present invention, a computer storage medium is provided for storing computer-readable instructions, which, when read by a computer, can execute the steps of the aforementioned method for real-time video processing in an augmented reality environment.

[0145] The above describes an embodiment of the present invention, but this embodiment is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Ordinary technicians in this field can also make more forms of equivalent embodiments based on the inspiration of this embodiment, all of which are protected by this embodiment.

[0146] It should be noted that:

[0147] The production of film and television concept drawings includes but is not limited to hand-drawing and computer production;

[0148] This method is not suitable for situations where the film and television concept images in a video are produced in different styles. For example, some film and television concept images are in a comic style, while others are in a realistic style. After such film and television concept image sequences are associated, it is difficult for the model to learn patterns.

[0149] The film and television concept map of the present invention does not include the concept map of independent individuals or objects in a local scene.

Claims

1. A real-time video processing method in an augmented reality environment, characterized in that: The following steps are involved: Step 101: sort the film and television concept images belonging to a film and television drama according to the development order of the described plot to obtain a concept image sequence; Step 102: sorting the scene texts of the storyboard script according to the sequence of shots to obtain a storyboard text sequence, where one storyboard text corresponds to the scene text of one shot; Sort the scene scripts in the order of the plot to obtain a scene text sequence, where one scene text corresponds to the entire text of a scene; Step 103: Graph structure data is constructed based on the concept image sequence, the storyboard text sequence, and the scene text sequence. The graph structure data includes basic data elements. One data element is associated with one concept image, storyboard text, or scene text. The association between two data elements is defined according to the following rules: The two data elements are associated with a film and television concept image and a storyboard text respectively. If the film and television concept image refers to the scene text and / or content text of the shot corresponding to the storyboard text during production, the association is defined as 1; otherwise, the association is defined as 0. The two data elements are associated with the film and television concept image and the scene text respectively. If the film and television concept image refers to the scene text when it is produced, the association is defined as 1, otherwise the association is defined as 0; If two data elements are both associated with storyboard texts, and the two storyboard texts contain more than B identical keywords, then the correlation is defined as 1, otherwise the correlation is defined as 0; If two data elements are both associated with scene texts and the two scene texts contain more than C identical keywords, the correlation is defined as 1, otherwise the correlation is defined as 0; Step 104: input the graph structure data into the trained augmented agent; The enhanced agent outputs an enhanced result of a concept image in the concept image sequence at each execution step.

2. The method for real-time video processing in an augmented reality environment according to claim 1, characterized in that: The value ranges of B and C are .

3. The method for real-time video processing in an augmented reality environment according to claim 1, wherein: Keywords are selected from scene texts and storyboard texts, and are manually selected as words that are considered important or the top N words in frequency.

4. The method for real-time video processing in an augmented reality environment according to claim 1, wherein: The enhanced agent includes an image encoding layer, a text encoding layer, a graph learning module, and a graph generation module. The image encoding layer is used to input a concept image and output image encoding features. The text encoding layer is used to input storyboard text and scene text respectively, and output storyboard text encoding features and scene text encoding features respectively. The graph learning module is used to input the feature representation of data elements and the association data of the data elements. The feature representation of data elements associated with film and television concept images is image coding features, the feature representation of data elements associated with storyboard text is storyboard text coding features, and the feature representation of data elements associated with scene text is scene text coding features. The i-th execution step for the augmented agent includes: Input the current graph structure data and output the data encoding features of the data element associated with the i-th concept image; Then, the data encoding features of the data element associated with the i-th concept image output by the graph learning module replace the feature representation of the i-th data element; The image encoding layer and text encoding layer only run in the first execution step, while the graph learning module runs in every execution step; The graph generation module includes a transformation layer and an upsampling layer. The transformation layer divides the data encoding features into three equal parts as three channels, and each channel is then tensored into a matrix form. The upsampling layer inputs the transformed data encoding features, and then performs upsampling to output the enhanced results of the conceptual image associated with the data element corresponding to the data encoding features.

5. The method for real-time video processing in an augmented reality environment according to claim 4, characterized in that: The graph learning module consists of D layers, where 6>D>0. The calculation formula for the lth layer is as follows: ； in is the attention weight of data element u to data element v, represents the set of data elements associated with data element v, Represents the first weight parameter of the lth layer, which is calculated by the following formula: ； a is a learnable attention vector, Represents vector concatenation operation, represents the second weight parameter of the lth layer; ； in and Represent the feature representation of the vth and uth data elements respectively.

6. The method for real-time video processing in an augmented reality environment according to claim 4, characterized in that: The enhanced results include resolution increase and / or style transfer relative to the original concept image.

7. The method for real-time video processing in an augmented reality environment according to claim 4, wherein: The image coding layer includes the CNN layer and the tensor layer. The CNN inputs the concept image and outputs the image features. The image features are expanded and spliced ​​row by row through the tensor layer to obtain the image coding features. The size of the image feature is a*b*3, 3 represents the number of channels, and the dimension of the image encoding feature is a*b*3; The dimension of the data encoding feature is a*b*3.

8. The method for real-time video processing in an augmented reality environment according to claim 1, wherein: Methods for training augmented agents include: Step 201: Initialize the experience pool, and randomly initialize the enhanced agent, the second network unit, the third network unit, and the fourth network unit; Step 202, resetting graph structure data; Step 203: Input graph structure data To enhance the agent to get the enhanced result of the t-th concept image ; The data encoding features of the data element associated with the t-th concept image output by the augmented agent replace the feature representation of the t-th data element to update the graph structure data ; Get image enhancement evaluation ; Enhanced evaluation comes from the evaluation of film and television professionals, and the range of the evaluation score is ; A score of 0 indicates that the evaluation of the video concept image after enhancement is consistent with that before enhancement; If the enhanced video concept image is better than the video concept image before enhancement, the score is higher, otherwise the score is lower; The termination condition done is that all concept images in the concept image sequence are enhanced; The experience Stored in the experience pool, one experience is a four-tuple or ; Step 204, randomly obtain N experiences from the experience pool; Input graph structure data To enhance the agent to get the enhanced result of the concept image ; Will and Enter the third network unit to get ; Input graph structure data Go to the second network unit to get the enhanced result of the concept image ; ； ； Represents input graph structure data and enhanced results of concept images The output of the fourth network unit; Represents input graph structure data and enhanced results of concept images The output of the fourth network unit; is a hyperparameter with a range of ; Find the loss and use the gradient descent method to update the network; ； ； pass Update the augmented agent by updating the third network unit; Step 205: every fixed number of network updates, update the second network unit so that its parameters are the same as those of the current augmented agent, and update the fourth network unit so that its parameters are the same as those of the current third network unit; Step 206, when the enhanced agent converges and the third network unit converges, the step is terminated or when the number of cycles of steps 203-205 reaches a set value, the step is terminated.

9. A computer storage medium, characterized in that It is used to store computer-readable instructions, which, when read by a computer, can execute the steps of the method for real-time video processing in an augmented reality environment as described in any one of claims 1 to 8.

10. A real-time video processing system in an augmented reality environment, characterized in that: include: An image processing module, which sorts the film and television concept images belonging to a film and television drama according to the development order of the described plot to obtain a concept image sequence; The text processing module sorts the scene texts of the storyboard script according to the sequence of shots to obtain a storyboard text sequence, where one storyboard text corresponds to the scene text of one shot; Sort the scene scripts in the order of the plot to obtain a scene text sequence, where one scene text corresponds to the entire text of a scene; The data processing module constructs graph structure data based on the concept image sequence, storyboard text sequence, and scene text sequence. The graph structure data includes basic data elements. Each data element is associated with a concept image, storyboard text, or scene text. The association between two data elements is defined according to rules. The image enhancement module inputs the graph structure data into the trained enhanced intelligent agent; the enhanced intelligent agent outputs the enhanced result of a concept image in the concept image sequence at each execution step.

Images