Image information processing method for short video production

Abstract

The invention relates to the technical field of image information processing, in particular to an image information processing method for short video production. According to the invention, each segment in a short video is analyzed and classified in real time by using a scene cognition algorithm; the scene cognition algorithm recognizes different scene types, and applies the most suitable image processing strategy for each scene; the definition of facial details and expressions is enhanced in a character close-up scene, the blurring and distortion of key points are reduced in a fast moving scene, and the optimal image quality is realized in each scene through dynamic processing; a digital twinning technology is introduced, a virtual model based on digital twinning is constructed, and virtual elements generated by AI are seamlessly integrated into an actual scene; physical characteristics of illumination, shadow, material reflection and the like in a real scene are simulated in real time by using a digital twin model, so that fusion between virtual content and a real background is more natural; the problem of image information processing in current short video production is solved.

Description

Technical Field

[0001] The present invention relates to the field of image information processing technology, and in particular to an image information processing method for short video production. Background Art

[0002] With the development of artificial intelligence, AI technology is used in the production of a large number of short videos. Although it improves production efficiency, it also brings about a general problem of quality decline. These low-quality AI-generated short videos usually lack details, have blurred images, and even have significant defects in content logic and aesthetics. They cannot meet the needs of users and high-end producers. Especially in creative scenarios with higher requirements, the limitations of existing AI production methods are more obvious.

[0003] In short video production, the processing requirements of different scenes are different, such as static scenes, dynamic scenes, close-up shots, and complex backgrounds. However, current AI technology usually uses a unified image processing algorithm to process these diverse scenes, lacking in-depth understanding of the scenes. This unified processing method leads to image blur and color distortion in scenes with rich details, and smear in fast-moving scenes, resulting in discontinuous images. To meet the needs of different scenes, existing methods optimize image quality by manually adjusting filters, compression parameters, and enhancement techniques. For example, in static scenes, high compression rates and edge enhancement techniques are used for optimization; in dynamic scenes, motion compensation algorithms are used to reduce blur, and low compression rates are used to retain the original image. More details; in addition, with the diversified development of short video content, more and more production needs involve the integration of virtual content and real scenes. However, the virtual elements generated by current AI generation technology often do not match the lighting, shadows, and materials of real scenes, resulting in unnatural visual effects. The audience can easily perceive the uncoordinated "fake" feeling, which seriously affects the overall quality and viewing experience of short videos. At present, traditional methods mainly rely on manual adjustment and post-production special effects processing to address the problem of "fake" feeling. By adjusting the lighting, shadows, and colors of virtual content to make it more matched with the real scene, post-production synthesis technology is used to integrate virtual elements with the real background. Post-production personnel perform fine post-production special effects processing to make the virtual content highly consistent with the actual shooting scene.

[0004] It can be seen that although the traditional method improves the image quality to a certain extent through manual setting and adjustment, manual adjustment of parameters is still time-consuming and laborious, and it is difficult to apply in real time in complex short video production. It lacks real scene recognition ability and cannot dynamically identify and adapt to changes in different scenes, resulting in unsatisfactory picture effects in scene transitions or fast-moving scenes, and cannot balance multiple visual factors. While improving image clarity, it will sacrifice color accuracy or produce visual artifacts; and for the "fake" sense of simulation, although manual adjustment and post-processing can improve the virtual-real combination effect to a certain extent, the manual adjustment process is also very time-consuming and requires a high level of technical skills from the operator. It is difficult to be widely used under the high-efficiency requirements of short video production, and it is better not to use AI. Secondly, this method relies on static or preset parameter adjustment. For dynamically changing scenes, such as moving light sources or changing camera angles, it is often difficult to quickly and accurately match, resulting in the combination of virtual and real in dynamic scenes is still not natural enough. Therefore, there is an urgent need for an image information processing method for short video production to solve such problems. Summary of the invention

[0005] To this end, the present invention provides an image information processing method for short video production, so as to overcome the problems in the prior art of lacking scene recognition ability, being unable to dynamically identify and adapt to changes in different scenes, having unsatisfactory picture effects in scene transitions or fast-motion scenes, having poor virtual-real combination effects, and having a very time-consuming manual adjustment process.

[0006] To achieve the above object, the present invention provides an image information processing method for short video production, comprising:

[0007] Step S1, video scene content analysis and classification, using scene recognition algorithm to analyze and classify each clip in the short video in real time; identifying different scene types, including close-ups of people, fast movements, and static backgrounds;

[0008] Step S2, adaptive image processing of the corresponding scene, based on the scene classification result of step S1, adaptively applying the corresponding image processing strategy for different scene types;

[0009] Step S3, virtual-real combination adjustment, introduce digital twin technology after image processing, integrate the virtual elements generated by AI into the actual scene, use the digital twin model to simulate the physical characteristics of the real scene in real time, and merge the virtual elements with the real background;

[0010] Step S4, feedback loop adjustment, uses a real-time feedback mechanism to continuously monitor the video output quality, and dynamically adjusts the previous processing links according to the feedback results.

[0011] Furthermore, scene classification and labeling are performed in step S1, and the scene recognition algorithm analyzes and classifies the video content in real time, generates scene labels, and divides them into character close-ups, fast movements, static, dynamic, etc., and assigns corresponding labels to each video clip.

[0012] Furthermore, in step S1, the real-time analysis and classification of each segment of the short video is as follows:

[0013] To extract video frame features, let the frame sequence of the short video be {I1,I2,...,I n}, where I i represents the i-th frame, where i represents 1, 2, ..., n;

[0014] Use the pre-trained deep convolutional neural network CNN to extract the feature vector F of each frame i , F i =φ(I i θ CNN ), where F i ∈R d , represents the feature vector of the i-th frame, with dimension d, φ(·; θ CNN ) represents the CNN feature extraction function, θ CNN Represents network parameters and extracts high-level features of the image to capture object, color, and edge information in the scene;

[0015] The video frame feature sequence is represented as {F1,F2,...,F n}, where n represents the number of video frame feature sequences, and time series feature fusion is performed. Long short-term memory network LSTM and gated recurrent unit GRU are used to model the frame features in time series to generate a time series feature vector H t , H t =ψ(F t ,H t 1;θ LSTM ), H t ∈R h represents the time series feature vector with dimension h, ψ(·; θ LSTM ) represents the LSTM unit, θ LSTM Represent network parameters; capture temporal dependencies between video frames, such as the continuity of character actions or the gradualness of background changes;

[0016] Input time series feature vector H t and other possible modal features, including audio features A t , text features T t , combine the features of different modes through the fusion network to generate a comprehensive feature vector C t , C t =γ(H t ,At ,T t θ Fusion ), where C t ∈R c represents the comprehensive feature vector with dimension c, γ(·; θ Fusion ) represents the fusion network function, θ Fusion Represents network parameters;

[0017] Use the fully connected layer and classifier to classify the feature vector and output the scene category of each video clip. in represents the predicted probability distribution of scene categories, k represents the number of scene categories, and W c ∈R k×c represents the classifier weight matrix, b c ∈R k represents the bias vector,

[0018] σ(·) is the softmax activation function, which achieves accurate classification of scenes through multimodal information fusion, distinguishing scenes such as close-ups of people, fast movements, and static backgrounds;

[0019] The classification results Converted into labels and annotated on the corresponding video clips to generate scene label sequences {L1,L2,...,L n}, where n represents the number of scene label sequences, Where L t is the scene category label to which the tth frame belongs; thereby mapping the scene classification result to a specific label.

[0020] Furthermore, dynamic processing is performed in step S2, and corresponding processing strategies are automatically applied according to scene labels. For close-up labeled segments of characters, facial details and expression clarity are enhanced first; for fast-motion labeled segments, motion blur and distortion are reduced; and the data output by the video scene content analysis and classification step is used to dynamically adjust image processing parameters to achieve the optimal processing effect in different scenarios.

[0021] Furthermore, the adaptive image processing method for the corresponding scene in step S2 is:

[0022] Input the scene label L from step S1 t And the video frame feature F t , L t Represents the scene label of the tth frame of the video, based on the scene label L t , select the processing strategy set corresponding to the scenario Each strategy set contains multiple optional image processing operations, namely in Indicates scene L tThe corresponding processing strategy set, M(L t ) represents the strategy mapping function, according to the scene label L t Provide corresponding treatment methods;

[0023] Perform strategy weight calculation and fusion, and input strategy set and video frame features F t , using a feature-based adaptive weight calculation model to assign weights to each strategy, combined with the feature vector F t , generate a weighted processing strategy, let P i represents the i-th processing strategy in the processing strategy set corresponding to the scenario, then where w i It is strategy P i The weight of i (F t ) represents the strategy P i The corresponding characteristic response function, α represents the smoothing coefficient, φ j (F t ) is the strategy P j The corresponding feature response function, j represents the number of processing strategies, and the input is the video frame feature vector F at a specific time t t , the output is a scalar value representing the feature vector F t Corresponding to strategy P j The response strength or adaptation degree of k represents the total number of image processing strategies available in a given scenario. The processing weights are dynamically allocated by calculating the response of the feature vector to different strategies.

[0024] Furthermore, the adaptive image processing method for the corresponding scene in step S2 also includes:

[0025] Process the close-up scene of the character, enhance the facial details, and input the feature vector F t , the corresponding weight w i , processing strategy set When L t When marked as "person close-up", detail enhancement processing strategies are preferred, including local contrast enhancement and super-resolution reconstruction, to improve facial details and expression clarity: Among them I t ′ represents the enhanced image, Ψ i (F t ,I t θ i ) represents a specific processing strategy P i For the input image I t The processing function, θ i represents the parameter set of the strategy;

[0026] For fast motion scenes, motion blur reduction is performed, and the input is the feature vector F t , the corresponding weight w i , processing strategy set When L t When marked as "fast motion", the motion blur correction strategy is preferred, including motion compensation based on optical flow and deblurring algorithm to reduce image blur:

[0027] in Represents the motion vector field estimated by optical flow, consisting of horizontal and vertical components u t , v t Composition, I′ t represents the deblurred image, Represents a specific image processing strategy P i Applied to image I given input t The processing function, Ψ i It includes a deblurring algorithm based on deconvolution and a motion compensation method based on optical flow vectors. k represents the total number of image processing strategies available in a given scene. The motion compensation strategy based on optical flow calculation estimates the motion vectors between adjacent frames, performs image alignment and reconstruction, and reduces motion blur. It combines the convolutional neural network (CNN) and the recurrent neural network (RNN) to further optimize the deblurring effect.

[0028] Perform adaptive parameter optimization and input the feature vector F t , scene label L t , the processed image I′ t , dynamically adjust the image processing parameters θ i , so that the processing results meet the predetermined quality goals (such as structural similarity, edge clarity, etc.): where θ i (t) Indicates the strategy P at the tth iteration i Parameters, η represents the learning rate, L(I′ t ,I t θ i ) represents the processed image I′ t and the original image I t The loss function uses the structural similarity index and gradient loss function;

[0029] Step S2 adaptively applies the corresponding image processing strategy automatically according to the scene label to optimize the visual effect in each scene, whether it is detail enhancement in close-ups of people or blur reduction in fast motion, to achieve the best processing effect.

[0030] Further, in step S3, a digital twin model is constructed based on the optimized image output by scene recognition and adaptive image processing;

[0031] As the scene changes dynamically (such as changes in camera angle or light source position), the digital twin model automatically adjusts the performance of virtual elements to keep it consistent with the real scene.

[0032] Furthermore, in step S3, a digital twin model is constructed:

[0033] The model parameters include: the optimized image sequence {I′1,I′2,...,I′ n}, scene feature vector {F1,F2,...,F n} and scene physical characteristic parameters {P1,P2,...,P m}, where n represents the number of image sequences and corresponding scene feature vectors, and m represents the number of scene physical characteristic parameters;

[0034] Build a digital twin model. in represents the digital twin simulation image at time t, Ω represents the entire hemisphere of the incident light direction, and f r (ω i ,ω o ; n) represents the bidirectional reflectance distribution function BRDF, describing the incident light direction ω i and the direction of the emitted light ω o The reflection relationship between the two surfaces, n represents the surface normal vector, L i (ω i ) represents the radiance of the incident light, n·ω i Represents the dot product of the incident light direction and the surface normal vector, dω i Represents differential solid angle; Based on the digital twin model, by simulating the light reflection characteristics in the scene, the lighting performance of the virtual elements in the scene is consistent with the actual environment;

[0035] Assume that the simulated image generated by the digital twin model is The actual scene image after optimization is I′ t And the virtual element image is V t , the virtual element V t With simulated image and the actual scene image I′ t Fusion is performed to generate the final composite image I' t ', I' t ' represents the final composite image at time t, α t and β tis the fusion coefficient at time t, which controls the weights of the virtual elements and simulated images in the final composite image respectively; by controlling α t and β t Realize seamless integration of virtual elements and actual scenes.

[0036] Furthermore, the virtual-real combination adjustment method in step S3 includes:

[0037] Dynamically adjust the virtual element performance, set the camera parameter to C t , including position, focal length, viewing angle; light source parameter L t , including the light source position, intensity, color, and initial state V of the virtual element t 0 ;

[0038] Dynamically adjust the performance of virtual elements as the camera angle or light source position changes: V t =argmin V L(V,C t ,L t θ V ),

[0039] Where V t is the state vector of the virtual element at time t, L(V,C t ,L t θ V ) is the loss function, which represents the virtual element state V and the current camera parameter C t And the light source parameter L t The inconsistency of V Represents a set of parameters that control the performance of a virtual element;

[0040] Real-time monitoring of video output quality indicators, including structural similarity SSIM, peak signal-to-noise ratio PSNR, synthetic images and loss function values, dynamically adjusts the digital twin model and virtual element performance.

[0041] Furthermore, the video output indicators monitored in real time in step S4 include: naturalness of virtual-real combination, image clarity, and color restoration;

[0042] If it is found that the processing effect in certain scenes does not meet expectations, the relevant algorithm parameters will be automatically adjusted and the problem fragments will be re-rendered; the feedback mechanism prompts the scene recognition algorithm and digital twin model to continuously optimize the processing strategy to form a closed loop.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention utilizes a scene recognition algorithm to perform real-time analysis and classification of each clip in a short video; the scene recognition algorithm identifies different scene types (such as close-ups of people, fast motion, static background, etc.) and applies the most suitable image processing strategy for each scene; the close-up scene of a person will enhance the clarity of facial details and expressions, and in the fast-motion scene, the focus will be on reducing blur and distortion, and the optimal image quality will be achieved in each scene through dynamic processing.

[0045] The present invention introduces digital twin technology, constructs a virtual model based on digital twins, and seamlessly integrates the virtual elements generated by AI into the actual scene. The digital twin model is used to simulate the physical properties of lighting, shadows, material reflection, etc. in the real scene in real time, making the fusion between virtual content and the real background more natural. The combination of virtual and real not only ensures the consistency of visual effects, but also improves the authenticity and detail expression of virtual elements.

[0046] In the present invention, the application of digital twin technology is based on the precise scene information provided by scene cognition, so that the combination of virtual and real scenes can dynamically adapt to different scene changes; for example, in scenes with mobile camera or light source changes, the digital twin model can adjust the presentation effect of virtual elements in real time to avoid visual discontinuity or incoordination.

[0047] The present invention adopts a real-time feedback mechanism to continuously monitor the quality of video output and adjust the parameters of the scene recognition algorithm and the digital twin model according to the feedback results; if it is detected in certain scenes that the combination of virtual and real is not natural enough or the image quality is degraded, the corresponding processing strategy is automatically adjusted and the part of the video is re-rendered.

[0048] It solves the current problem of image information processing in short video production, combines the dynamic adaptability of scene cognition with the high-precision virtual-real fusion of digital twin technology, improves the visual effects of short videos, and enhances the realism and consistency of the content. It can effectively meet the needs of high-end producers for high-quality and high-realism short videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of the flow of the image information processing method for short video production of the present invention. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0051] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0052] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0053] Example 1

[0054] See also Figure 1 The present invention provides an image information processing method for short video production, comprising:

[0055] Step S1, video scene content analysis and classification, using scene recognition algorithm to analyze and classify each clip in the short video in real time; identifying different scene types, including close-ups of people, fast movements, and static backgrounds;

[0056] In step S1, scene classification and labeling are performed. The scene recognition algorithm analyzes and classifies the video content in real time, generates scene labels, and divides them into character close-up, fast motion, static, dynamic, etc., and labels each video clip accordingly;

[0057] In step S1, the real-time analysis and classification of each segment of the short video is as follows:

[0058] To extract video frame features, assume that the frame sequence of the short video is {I1,I2,…,I n}, where I i represents the i-th frame, where i represents 1, 2, ..., n;

[0059] Use the pre-trained deep convolutional neural network CNN to extract the feature vector F of each frame i , F i =φ(I i θ CNN ), where F i ∈R d , represents the feature vector of the i-th frame, with dimension d, φ(·; θ CNN ) represents the CNN feature extraction function, θ CNNRepresents network parameters and extracts high-level features of the image to capture object, color, and edge information in the scene;

[0060] The video frame feature sequence is represented as {F1,F2,...,F n}, where n represents the number of video frame feature sequences, and time series feature fusion is performed. Long short-term memory network LSTM and gated recurrent unit GRU are used to model the frame features in time series to generate a time series feature vector H t , H t =ψ(F t ,H t 1;θ LSTM ), H t ∈R h represents the time series feature vector with dimension h, ψ(·; θ LSTM ) represents the LSTM unit, θ LSTM Represent network parameters; capture temporal dependencies between video frames, such as the continuity of character actions or the gradualness of background changes;

[0061] Input time series feature vector H t and other possible modal features, including audio features A t , text features T t , combine the features of different modes through the fusion network to generate a comprehensive feature vector C t , C t =γ(H t ,A t ,T t θ Fusion ), where C t ∈R c represents the comprehensive feature vector, with dimension c,

[0062] γ(·;θ Fusion ) represents the fusion network function, θ Fusion Represents network parameters;

[0063] Use the fully connected layer and classifier to classify the feature vector and output the scene category of each video clip in represents the predicted probability distribution of scene categories, k represents the number of scene categories, and W c ∈R k×c represents the classifier weight matrix, b c ∈R k represents the bias vector,

[0064] σ(·) is the softmax activation function, which achieves accurate classification of scenes through multimodal information fusion, distinguishing scenes such as close-ups of people, fast movements, and static backgrounds;

[0065] The classification results Converted into labels and annotated on the corresponding video clips to generate scene label sequences {L1,L2,...,L n}, where n represents the number of scene label sequences, Where L t is the scene category label to which the tth frame belongs; thus mapping the scene classification result to a specific label;

[0066] Specifically, the scene recognition algorithm analyzes and classifies each clip in the video in real time, accurately identifying different scenes such as close-ups of people, fast movements, static backgrounds, etc.; fine-grained classification enables the processing strategy to accurately match various scenes, avoiding the problem of image quality degradation caused by unified processing methods.

[0067] Step S2, adaptive image processing of the corresponding scene, based on the scene classification result of step S1, adaptively applying the corresponding image processing strategy for different scene types;

[0068] In step S2, dynamic processing is performed, and corresponding processing strategies are automatically applied according to scene labels. For close-up labeled segments of characters, facial details and expression clarity are enhanced first; for fast-motion labeled segments, motion blur and distortion are reduced; the data output from the video scene content analysis and classification step is used to dynamically adjust image processing parameters to achieve the optimal processing effect in different scenes;

[0069] The adaptive image processing method for the corresponding scene in step S2 is:

[0070] Input the scene label L from step S1 t And the video frame feature F t , L t Represents the scene label of the tth frame of the video, based on the scene label L t , select the processing strategy set corresponding to the scenario Each strategy set contains multiple optional image processing operations, namely in Indicates scene L t The corresponding processing strategy set, M(L t ) represents the strategy mapping function, according to the scene label L t Provide corresponding treatment methods;

[0071] Perform strategy weight calculation and fusion, and input strategy set and video frame features F t , using a feature-based adaptive weight calculation model to assign weights to each strategy, combined with the feature vector F t , generate a weighted processing strategy, let P irepresents the i-th processing strategy in the processing strategy set corresponding to the scenario, then where w i It is strategy P i The weight of i (F t ) represents the strategy P i The corresponding characteristic response function, α represents the smoothing coefficient, φ j (F t ) is the strategy P j The corresponding feature response function, j represents the number of processing strategies, and the input is the video frame feature vector F at a specific time t t , the output is a scalar value representing the feature vector F t Corresponding to strategy P j The response strength or degree of adaptation, k represents the total number of image processing strategies available in a given scenario, and the processing weights are dynamically allocated by calculating the response of the feature vector to different strategies;

[0072] Process the close-up scene of the character, enhance the facial details, and input the feature vector F t , the corresponding weight w i , processing strategy set When L t When marked as "person close-up", detail enhancement processing strategies are preferred, including local contrast enhancement and super-resolution reconstruction, to improve facial details and expression clarity: Among them I t ′ represents the enhanced image, Ψ i (F t ,I t θ i ) represents a specific processing strategy P i For the input image I t The processing function, θ i represents the parameter set of the strategy;

[0073] For fast motion scenes, motion blur reduction is performed, and the input is the feature vector F t , the corresponding weight w i , processing strategy set When L t When marked as "fast motion", the motion blur correction strategy is preferred, including motion compensation based on optical flow and deblurring algorithm to reduce image blur:

[0074] in Represents the motion vector field estimated by optical flow, consisting of horizontal and vertical components u t , v t Composition, I′ t represents the deblurred image, Represents a specific image processing strategy P i Applied to image I given input t The processing function, Ψ i It includes a deblurring algorithm based on deconvolution and a motion compensation method based on optical flow vectors. k represents the total number of image processing strategies available in a given scene. The motion compensation strategy based on optical flow calculation estimates the motion vectors between adjacent frames, performs image alignment and reconstruction, and reduces motion blur. It combines the convolutional neural network (CNN) and the recurrent neural network (RNN) to further optimize the deblurring effect.

[0075] Perform adaptive parameter optimization and input the feature vector F t , scene label L t , the processed image I′ t , dynamically adjust the image processing parameters θ i , so that the processing results meet the predetermined quality goals (such as structural similarity, edge clarity, etc.): where θ i (t) Indicates the strategy P at the tth iteration i Parameters, η represents the learning rate, L(I′ t ,I t θ i ) represents the processed image I′ t and the original image I t The loss function uses the structural similarity index and gradient loss function;

[0076] Step S2 adaptively applies the corresponding image processing strategy automatically according to the scene label to optimize the visual effect in each scene, whether it is detail enhancement in a close-up of a person or blur reduction in fast motion, to achieve the best processing effect;

[0077] Specifically, in the adaptive image processing stage, the processing strategy is dynamically adjusted according to the scene label. For close-up scenes of characters, the algorithm prioritizes enhancing facial details and expression clarity, while in fast-motion scenes, it reduces motion blur and distortion to optimize picture quality. The adaptive processing method not only improves the clarity and expressiveness of the image, but also can dynamically allocate weights to greatly improve the overall picture quality.

[0078] Step S3, virtual-real combination adjustment, introduce digital twin technology after image processing, integrate the virtual elements generated by AI into the actual scene, use the digital twin model to simulate the physical characteristics of the real scene in real time, and merge the virtual elements with the real background;

[0079] In step S3, a digital twin model is constructed based on the optimized image output by scene recognition and adaptive image processing;

[0080] As the scene changes dynamically (such as changes in camera angle or light source position), the digital twin model automatically adjusts the performance of virtual elements to keep it consistent with the real scene;

[0081] In step S3, the digital twin model is constructed:

[0082] The model parameters include: the optimized image sequence {I′1,I′2,...,I′ n}, scene feature vector {F1,F2,...,F n} and scene physical characteristic parameters {P1,P2,...,P m}, where n represents the number of image sequences and corresponding scene feature vectors, and m represents the number of scene physical characteristic parameters;

[0083] Build a digital twin model. in represents the digital twin simulation image at time t, Ω represents the entire hemisphere of the incident light direction, and f r (ω i ,ω o ; n) represents the bidirectional reflectance distribution function BRDF, describing the incident light direction ω i and the direction of the emitted light ω o The reflection relationship between the two surfaces, n represents the surface normal vector, L i (ω i ) represents the radiance of the incident light, n·ω i Represents the dot product of the incident light direction and the surface normal vector, dω i Represents differential solid angle; Based on the digital twin model, by simulating the light reflection characteristics in the scene, the lighting performance of the virtual elements in the scene is consistent with the actual environment;

[0084] Assume that the simulated image generated by the digital twin model is The actual scene image after optimization is I′ t And the virtual element image is V t , the virtual element V t With simulated image and the actual scene image I′ t Fusion is performed to generate the final composite image I' t ', I' t ' represents the final composite image at time t, α t and β t is the fusion coefficient at time t, which controls the weights of the virtual elements and simulated images in the final composite image respectively; by controlling α t and β t Achieve seamless integration of virtual elements and actual scenes;

[0085] The virtual-real combination adjustment method in step S3 includes:

[0086] Dynamically adjust the virtual element performance, set the camera parameter to C t , including position, focal length, viewing angle; light source parameter L t , including the light source position, intensity, color, and initial state V of the virtual element t 0 ;

[0087] Dynamically adjust the performance of virtual elements as the camera angle or light source position changes: V t =argmin V L(V,C t ,L t θ V ),

[0088] Where V t is the state vector of the virtual element at time t, L(V,C t ,L t θ V ) is the loss function, which represents the virtual element state V and the current camera parameter C t And the light source parameter L t The inconsistency of V Represents a set of parameters that control the performance of a virtual element;

[0089] Real-time monitoring of video output quality indicators, including structural similarity SSIM, peak signal-to-noise ratio PSNR, synthetic images and loss function values, dynamically adjusting the digital twin model and virtual element performance;

[0090] Specifically, by simulating the physical properties of real scenes (such as lighting, shadows, and material reflections), AI-generated virtual elements are seamlessly integrated into actual scenes. As the camera angle or light source position changes, the digital twin model can dynamically adjust the performance of virtual elements so that they are always consistent with the real scene. It effectively eliminates the common "fake feeling" when traditional virtual elements are integrated with real scenes, enhancing the immersion and visual consistency of the video.

[0091] Step S4, feedback loop adjustment, using a real-time feedback mechanism to continuously monitor the video output quality, and dynamically adjust the previous processing steps according to the feedback results;

[0092] The video output indicators monitored in real time in step S4 include: naturalness of virtual-real combination, image clarity, and color restoration;

[0093] If it is found that the processing effect in some scenes does not meet expectations, the relevant algorithm parameters are automatically adjusted and the problem fragments are re-rendered; the feedback mechanism prompts the scene recognition algorithm and digital twin model to continuously optimize the processing strategy to form a closed loop;

[0094] Specifically, by real-time monitoring of video output quality indicators such as structural similarity, image clarity and color reproduction, the processing strategy is automatically adjusted and re-rendering is performed when problems are found, forming a closed-loop optimization process.

[0095] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A method for processing image information for short video production, characterized in that: include: Step S1, video scene content analysis and classification, analyzing and classifying each clip in the short video, identifying different scene types, including close-ups of people, fast movements, and static backgrounds; Step S2, adaptive image processing of the corresponding scene, based on the scene classification result of step S1, adaptively applying the corresponding image processing strategy for different scene types; Step S3, virtual-real combination adjustment, integrating the virtual elements generated by AI into the actual scene, using the digital twin model to simulate the physical characteristics of the real scene in real time, and integrating the virtual elements with the real background of the video scene; Step S4, feedback loop adjustment, continuously monitors the video output quality, and dynamically adjusts the processing methods of step S2 and step S3 according to the feedback results.

2. The image information processing method for short video production according to claim 1, characterized in that: In step S1, scene classification and labeling are performed, and a scene recognition algorithm is used to analyze and classify video content in real time, generate corresponding scene labels, and divide the scene labels into character close-up, fast motion, static, and dynamic.

3. The image information processing method for short video production according to claim 2, characterized in that: In step S1, the method of real-time analysis and classification of each segment of the short video is as follows: To extract video frame features, assume that the frame sequence of the short video is {I1,I2,…,I n }, where I i represents the i-th frame, where i represents 1, 2, ..., n; Use the pre-trained deep convolutional neural network CNN to extract the feature vector F of each frame i , F i =φ(I i θ CNN ), where F i ∈R d , represents the feature vector of the i-th frame, with dimension d, φ(·; θ CNN ) represents the CNN feature extraction function, θ CNN Represents network parameters; The video frame feature sequence is represented as {F1,F2,...,F n }, where n represents the number of video frame feature sequences, and time series feature fusion is performed. Long short-term memory network LSTM and gated recurrent unit GRU are used to model the frame features in time series to generate a time series feature vector H t , H t =ψ(F t ,H t1 θ LSTM ), H t ∈R h represents the time series feature vector with dimension h, ψ(·; θ LSTM ) represents the LSTM unit, θ LSTM Represents network parameters; Input time series feature vector H t And modal features, modal features include audio features A t , text features T t , combining different modal features to generate a comprehensive feature vector C t , C t =γ(H t ,A t ,T t θ Fusion ), where C t ∈R c represents the comprehensive feature vector with dimension c, γ(·; θ Fusion ) represents the fusion network function, θ Fusion Represents network parameters; Use the fully connected layer and classifier to classify the feature vector and output the scene category of each video clip. in, represents the predicted probability distribution of scene categories, k represents the number of scene categories, and W c ∈R k×c represents the classifier weight matrix, b c ∈R k represents the bias vector, σ(·) represents the softmax activation function; The classification results Converted into labels and annotated on the corresponding video clips to generate scene label sequences {L1,L2,...,L n }, where n represents the number of scene label sequences, Where L t is the scene category label to which the tth frame belongs.

4. The image information processing method for short video production according to claim 3, characterized in that: In step S2, dynamic processing is performed and the corresponding processing strategy is automatically applied according to the scene label: For close-up clips of people, give priority to enhancing facial details and expression clarity; Reduce motion blur and distortion for fast-motion tag clips.

5. The image information processing method for short video production according to claim 4, characterized in that: The adaptive image processing method for the corresponding scene in step S2 includes: Input the scene label L from step S1 t And the video frame feature F t , L t Represents the scene label of the tth frame of the video, based on the scene label L t , select the processing strategy set corresponding to the scenario Each strategy set contains multiple optional image processing operations, namely in Indicates scene L t The corresponding processing strategy set, M(L t ) represents the strategy mapping function, according to the scene label L t Provide corresponding treatment methods; Using the feature-based adaptive weight calculation model, we assign weights to each strategy and combine it with the feature vector F t , generate a weighted processing strategy, let P i represents the i-th processing strategy in the processing strategy set corresponding to the scenario, then where w i It is strategy P i The weight of i (F t ) represents the strategy P i The corresponding characteristic response function, α represents the smoothing coefficient, φ j (F t ) is the strategy P j The corresponding feature response function, j represents the number of processing strategies, and the input is the video frame feature vector F at a specific time t t , the output is a scalar value representing the feature vector F t Corresponding to strategy P j k represents the total number of image processing strategies available in a given scenario.

6. The image information processing method for short video production according to claim 5, characterized in that: The adaptive image processing method for the corresponding scene in step S2 also includes: Process the close-up scene of a person and enhance the facial details. Suppose the input feature vector F t , the corresponding weight w i , processing strategy set When L t When marked as "person close-up", the detail enhancement processing strategy is preferred, including local contrast enhancement and super-resolution reconstruction. Where I′ t represents the enhanced image, Ψ i (F t ,I t θ i ) represents a specific processing strategy P i For the input image I t The processing function, θ i represents the parameter set of the strategy, i represents the number of processing strategies, k represents the total number of processing strategies, and t represents the video frame; For fast motion scenes, motion blur reduction is performed. Let the input be the feature vector F t , the corresponding weight w i , processing strategy set When L t When marked as "fast motion", select the motion blur correction strategy, including motion compensation based on optical flow and deblurring algorithm: in Represents the motion vector field estimated by optical flow, consisting of horizontal and vertical components u t , v t Composition, I′ t represents the deblurred image, Represents the image processing strategy P i Applied to image I given input conditions t The processing function, Ψ i Including deblurring algorithm based on deconvolution and motion compensation method based on optical flow vector; Perform adaptive parameter optimization and input the feature vector F t , scene label L t , the processed image I′ t , dynamically adjust the image processing parameters θ i , so that the processing results meet the predetermined quality goals: in Indicates the strategy P at the tth iteration i Parameters, η represents the learning rate, L(I′ t ,I t θ i ) represents the processed image I′ t and the original image I t The loss function of .

7. The image information processing method for short video production according to claim 6, characterized in that: In step S3, the virtual-real combination adjustment further includes: Based on the image optimized by the image processing strategy outputted by step S2, a digital twin model is constructed.

8. The image information processing method for short video production according to claim 7, characterized in that: The digital twin model is constructed in step S3 as follows: The model parameters include: the optimized image sequence {I′1,I′2,...,I′n}, the scene feature vector {F1,F2,...,F n } and scene physical characteristic parameters {P1,P2,...,P m }, where n represents the number of image sequences and corresponding scene feature vectors, and m represents the number of scene physical characteristic parameters; Build a digital twin model. in represents the digital twin simulation image at time t, Ω represents the hemisphere of the incident light direction, and f r (ω i ,ω o ; n) represents the bidirectional reflectance distribution function BRDF, which is used to describe the direction of the incident light ω i and the direction of the emitted light ω O The reflection relationship between the two surfaces, n represents the surface normal vector, L i (ω i ) represents the radiance of the incident light, n·ω i Represents the dot product of the incident light direction and the surface normal vector, dω i represents the differential solid angle; Assume that the simulated image generated by the digital twin model is The actual scene image after optimization is I′ t And the virtual element image is V t , the virtual element V t With simulated image and the actual scene image I′ t Fusion is performed to generate the final composite image I″ t , Where I″ t represents the final composite image at time t, α t and β t are the fusion coefficients at time t, which control the weights of the virtual elements and simulated images in the final synthesized image respectively.

9. The image information processing method for short video production according to claim 8, characterized in that: The virtual-real combination adjustment method in step S3 includes: Dynamically adjust the virtual element performance, set the camera parameter to C t , light source parameter L t , and the initial state of the virtual element Dynamically adjust virtual element performance as the camera angle changes: V t =argmin V L(V,C t ,L t θ V ), where V t represents the state vector of the virtual element at time t, L(V,C t ,L t θ V ) is the loss function, which represents the virtual element state V and the current camera parameter C t And the light source parameter L t The inconsistency of V Represents a set of parameters that control the behavior of a virtual element.

10. The image information processing method for short video production according to claim 9, characterized in that: The video output quality continuously monitored in step S4 includes: the naturalness of the combination of virtuality and reality, image clarity, and color reproduction.

Images