Computer video image generation method and system

Abstract

The invention is suitable for an image generation technology, and provides a computer video image generation method and system, and the system comprises a frame extraction and analysis module, a light source recognition and analysis module, an illumination model construction module, a shadow recognition and calculation module, and an object tracking and frame insertion generation module. By systematically extracting video frames and identifying objects, backgrounds, light sources and shadow areas of adjacent frames, the method can accurately capture dynamic changes in a scene, and ensures the coherence between inserted frames and original video content. The comprehensive frame analysis provides a solid foundation for subsequent illumination and shadow processing, so that the inserted frame can be more naturally fused into the whole video stream. The accurate illumination analysis not only improves the stereoscopic impression and reality sense of the objects, but also enhances the interaction between the objects, so that the whole scene is more unified and coordinated visually.

Description

Technical Field

[0001] The present invention belongs to the technical field of image generation, and in particular relates to a computer video image generation method and system. Background Art

[0002] Image generation technology is an important branch of computer vision and artificial intelligence, which aims to create images from scratch or generate new variants from existing images. The current image generation based on interpolation technology is to improve the visual quality and smoothness of the video. By generating new intermediate frames between the original video frames, interpolation technology can significantly increase the frame rate, so that low-frame-rate videos can be converted to higher frame rates and show smoother picture effects. This is especially important in fast-moving scenes because it helps reduce motion blur and improves the audience's viewing experience. In addition, interpolation technology can effectively reduce motion artifacts caused by insufficient frame rate, making the movement of objects in dynamic scenes more natural, thereby improving visual effects.

[0003] The generation of shadows depends on the direction and intensity of the light source and the shape of the occluder. However, many existing technologies fail to fully consider these dynamic factors when generating interpolated frames. For example, in indoor scenes, the angle and intensity of light outside the window will change over time. If the lighting is not accurately adjusted during interpolation, the lighting intensity of indoor objects may not match that of the surrounding frames, making these objects appear too dark or too bright in the interpolated frames, thus destroying the realism of the scene.

[0004] In dynamic performance scenes, lighting conditions often change with the action of the scene, such as the movement of lights or the adjustment of their intensity. If these lighting changes are not updated synchronously during the interpolation process, the shadows and brightness of the inserted frames may be inconsistent with the previous and next frames, causing the shadows of objects on the screen to be unstable or jump between different frames. Summary of the invention

[0005] The purpose of the present invention is to provide a computer video image generation method, aiming to solve the technical problems existing in the prior art identified in the background technology.

[0006] The present invention is implemented as follows: a method for generating a computer video image, the method comprising:

[0007] Extract all frames from the video, determine the adjacent frames that need to be inserted, take the adjacent frames as the extraction center, and extract the video frames on both sides of the adjacent frames;

[0008] By calculating the RGB value of each pixel, analyzing the brightness and color distribution, identifying the edge of the light source, and identifying the position and intensity of the light source in the scene:

[0009] According to the position of the light source, taking into account the factors of ambient light, diffuse reflection and specular reflection, a lighting model is constructed to calculate the light intensity for each object and analyze the effects of light on different objects;

[0010] Analyze the lighting changes between adjacent frames, identify the shadow area, and compare it with the non-shadow area to identify the available shadow area. Based on the light source position, calculate the shadow shape and position of each object. According to the height of each object and the light source angle, set the shadow length and shape corresponding to each object, and generate the shadow information of the inserted frame.

[0011] Track the motion trajectory of each object between adjacent frames, and take the middle position of the object between adjacent frames as the position of the corresponding object in the inserted frame as the smooth motion trajectory of the object, and adjust the shadow information of the inserted frame according to the lighting status of the adjacent frames to obtain the final image of the inserted frame.

[0012] As a further solution of the present invention, the step of determining the adjacent frames that need to be interpolated, taking the adjacent frames as extraction centers, and extracting video frames on both sides of the adjacent frames specifically includes:

[0013] Read the video file, set the extraction frequency to extract frames from the video, and store them in the form of an array;

[0014] Analyze the video frames in the array, identify the existing objects, and combine several adjacent frames to determine whether the object movements in several adjacent frames are temporally coherent. If there is discontinuous object change between two adjacent frames, the two adjacent frames are defined as adjacent frames that need to be interpolated.

[0015] Set the extraction duration, perform secondary extraction on the adjacent frames that need to be inserted, take the adjacent frames that need to be inserted as the time center, extend the same extraction duration to both sides, extract all video frames covered by the duration, and store them as an extraction group.

[0016] As a further solution of the present invention, the analysis of brightness and color distribution, identification of light source edges, and identification of light source positions and intensities in a scene specifically include:

[0017] Analyze the video frames in the extraction group, traverse each pixel of the image in each video frame, extract the RGB value, and convert it to a brightness value ;

[0018] ;

[0019] in Represents the red channel value of the pixel. Represents the green channel value of the pixel. Represents the blue channel value of the pixel;

[0020] Generate a color histogram based on the extracted RGB values ​​and count the number of pixels of each color;

[0021] According to the edge detection algorithm, the edges in the image are identified, the intensity change threshold is set, the edge information is extracted, and the area where the intensity change exceeds the threshold is identified, which is defined as the potential light source position;

[0022] Identify the point with the highest brightness value among the potential light source positions as the light source position, and calculate the average brightness value of the pixels around the light source position as the light source intensity;

[0023] Determine the light source type based on the light source intensity and light source position distribution.

[0024] As a further solution of the present invention, the construction of the illumination model, calculation of the illumination intensity for each object, and analysis of the illumination effects on different objects specifically include:

[0025] Establish a lighting model and set the ambient light intensity value according to the light source intensity , used to simulate the scattered light in the scene and calculate the impact of ambient light on the surface of each object;

[0026] For adjacent objects, consider the interaction between occlusion and lighting, evaluate the changes in shadow and lighting effects, and adjust the lighting vector when the object changes its position and direction relative to the light source. Adjust the direction and intensity of

[0027] The calculated illumination model parameters and illumination intensity information are output to the data structure and stored.

[0028] As a further solution of the present invention, the influence of the ambient light on the surface of each object is calculated, specifically:

[0029] Calculate the illumination vector from the light source to the surface of the object, obtain the normal vector of the surface of the object, and calculate the diffuse illumination intensity:

[0030] ;

[0031] in, is the diffuse reflected light intensity, is the light source intensity, is the normal vector, , is the light vector;

[0032] in, ;

[0033] Calculate the viewing angle vector between the video image acquisition point and the object surface point, and calculate the reflection vector through the illumination vector and normal vector:

[0034] ;

[0035] Calculating specular intensity :

[0036] ;

[0037] in, is the mirror reflection coefficient, take 0.5, is the viewing angle vector;

[0038] Calculate global ambient light intensity :

[0039] .

[0040] As a further solution of the present invention, the generating of the shadow information of the inserted frame specifically includes:

[0041] According to the brightness distribution of adjacent frames, the brightness threshold for distinguishing shadow areas from non-shadow areas is automatically set, the brightness matrix of the current frame is traversed, all pixel areas with brightness below the threshold are identified, and marked as potential shadow areas;

[0042] Conduct connectivity analysis on potential shadow areas and merge adjacent potential shadow areas to form a continuous shadow area.

[0043] According to the results of the light source position and light source type, the shadow length is calculated based on the height of the object, the height and direction of the light source:

[0044] ;

[0045] in, represents the shadow length, Indicates the height of the object. Indicates the light angle;

[0046] Generate the shadow shape of each object inserted into the frame according to the direction and distance of the light source, and refine the shadow outline;

[0047] According to the light intensity, distance and ambient light intensity value, the shadow transparency and shadow brightness are adjusted to obtain the final shadow information.

[0048] As a further solution of the present invention, the tracking of the motion trajectory of each object between adjacent frames, taking the middle position of the object between adjacent frames as the position of the corresponding object in the inserted frame as the smooth motion trajectory of the object, and adjusting the shadow information of the inserted frame according to the lighting state of the adjacent frames to obtain the final image of the inserted frame specifically includes:

[0049] Using the identified existing objects, extract the list of objects that need to be tracked in the inserted frame;

[0050] For each object, calculate the position difference between the object in the adjacent frames on both sides of the inserted frame to obtain the object motion vector:

[0051] ;

[0052] in, is the object's motion vector, Indicates the center position of the object in the frame after the insertion frame. Indicates the center position of the object in the frame before the inserted frame;

[0053] Based on the position of the object in the adjacent frames, calculate its middle position in the inserted frame:

[0054] ;

[0055] in, is the middle position of the object in the inserted frame;

[0056] Extract the lighting status of adjacent frames and verify and adjust the final shadow information;

[0057] The intermediate positions, lighting states, and shadow information of all objects in the inserted frames are synthesized into one image to generate the final inserted frame image.

[0058] Another object of the present invention is to provide a computer video image generation system, the system comprising:

[0059] The frame extraction and analysis module is used to extract all frames from the video, determine the adjacent frames that need to be inserted, take the adjacent frames as the extraction center, and extract the video frames on both sides of the adjacent frames;

[0060] The light source identification and analysis module is used to analyze the brightness and color distribution, identify the edge of the light source, and identify the position and intensity of the light source in the scene by calculating the RGB value of each pixel:

[0061] The illumination model building module is used to build an illumination model based on the position of the light source, taking into account ambient light, diffuse reflection and specular reflection factors, calculate the illumination intensity for each object, and analyze the illumination effects on different objects;

[0062] Shadow recognition and calculation module, used to analyze the illumination changes between adjacent frames, identify the shadow area, compare it with the non-shadow area, identify the available shadow area, calculate the shadow shape and position of each object based on the light source position, set the shadow length and shape corresponding to each object according to the height of each object and the light source angle, and generate the shadow information of the inserted frame;

[0063] The object tracking and interpolation frame generation module is used to track the motion trajectory of each object between adjacent frames, and take the middle position of the object between adjacent frames as the position of the corresponding object in the interpolation frame as the smooth motion trajectory of the object, and adjust the shadow information of the interpolation frame according to the lighting status of the adjacent frames to obtain the final image of the interpolation frame.

[0064] As a further solution of the present invention, the frame extraction and analysis module includes:

[0065] The video reading unit is used to read the video file, set the extraction frequency to extract frames from the video, and store them in the form of an array;

[0066] The object motion continuity judgment unit is used to analyze the video frames in the array, identify the existing objects, and combine several adjacent frames to judge whether the object motion in several adjacent frames is temporally coherent. If there is discontinuous object change between two adjacent frames, the two adjacent frames are defined as adjacent frames that need to be interpolated.

[0067] The secondary extraction time setting unit is used to set the extraction time, perform secondary extraction on adjacent frames that need to be inserted, take the adjacent frames that need to be inserted as the time center, extend the same extraction time to both sides, extract all video frames covered by the time, and store them as an extraction group.

[0068] As a further solution of the present invention, the light source identification and analysis module includes:

[0069] A pixel RGB value extraction unit is used to analyze the video frames in the extraction group, traverse each pixel of the image in each video frame, extract the RGB value, and convert it into a brightness value;

[0070] A color histogram generating unit, used to generate a color histogram according to the extracted RGB values ​​and count the number of pixels of each color;

[0071] An edge detection and intensity change analysis unit, which is used to identify edges in the image based on an edge detection algorithm, set an intensity change threshold, extract edge information, and identify areas where the intensity change exceeds the threshold, which are defined as potential light source locations;

[0072] A light source position identification unit, used to identify the point with the highest brightness value among the potential light source positions as the light source position, and calculate the average brightness value of the pixels around the light source position as the light source intensity;

[0073] The light source type determination unit is used to determine the light source type based on the light source intensity and the light source position distribution.

[0074] The beneficial effects of the present invention are:

[0075] By systematically extracting video frames and identifying objects, backgrounds, light sources, and shadow areas in adjacent frames, this method can accurately capture dynamic changes in the scene and ensure the coherence between the inserted frames and the original video content. This comprehensive frame analysis provides a solid foundation for subsequent lighting and shadow processing, allowing the inserted frames to be more naturally integrated into the overall video stream.

[0076] This method can accurately calculate the light intensity of each object by building a detailed lighting model, taking into account multiple factors such as ambient light, diffuse reflection and specular reflection. This precise lighting analysis not only enhances the three-dimensionality and realism of objects, but also enhances the interaction between objects, making the entire scene more visually unified and coordinated. In addition, the process of automatically identifying and adjusting shadow information ensures that the shape, position and intensity of the shadow can adapt to changes in the light source, thereby effectively eliminating the phenomenon of objects "floating" on the background.

[0077] By tracking the motion trajectory of each object between adjacent frames, this method achieves a smooth transition of objects in the inserted frames, ensuring the fluidity and naturalness of dynamic scenes. The shadow information is adjusted in combination with the lighting status of adjacent frames, further enhancing the realism of the inserted frames. This comprehensive lighting and shadow processing method makes the generated inserted frames not only more visually harmonious, but also provides a more pleasant viewing experience for the audience. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A flowchart of a method for generating a computer video image provided by an embodiment of the present invention;

[0079] Figure 2 A flowchart for determining adjacent frames that need to be inserted, taking the adjacent frames as extraction centers, and extracting video frames on both sides of the adjacent frames for identification and analysis provided by an embodiment of the present invention;

[0080] Figure 3 A flowchart for analyzing brightness and color distribution, identifying light source edges, and identifying light source positions and intensities in a scene provided by an embodiment of the present invention;

[0081] Figure 4 A flowchart for constructing a lighting model, calculating the lighting intensity for each object, and analyzing the lighting effects on different objects provided by an embodiment of the present invention;

[0082] Figure 5 A flowchart of generating shadow information of an inserted frame provided by an embodiment of the present invention;

[0083] Figure 6 A flowchart of obtaining a final image of an inserted frame provided by an embodiment of the present invention;

[0084] Figure 7A structural block diagram of a computer video image generation system provided by an embodiment of the present invention;

[0085] Figure 8 A structural block diagram of a frame extraction and analysis module provided in an embodiment of the present invention;

[0086] Fig. 9 This is a structural block diagram of a light source identification and analysis module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0088] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0089] Figure 1 A flowchart of a method for generating a computer video image provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0090] S100, extracting all frames from the video, determining adjacent frames that need to be inserted, taking the adjacent frames as extraction centers, and extracting video frames on both sides of the adjacent frames;

[0091] like Figure 2 As shown, the step of determining the adjacent frames that need to be inserted, taking the adjacent frames as extraction centers, and extracting video frames on both sides of the adjacent frames specifically includes:

[0092] S110, reading a video file, setting an extraction frequency to extract frames from the video, and storing them in the form of an array;

[0093] S120, analyzing the video frames in the array, identifying the existing objects, and combining several adjacent frames to determine whether the motion of the objects in the adjacent frames is temporally coherent. If there is an intermittent object change between two adjacent frames, the two adjacent frames are defined as adjacent frames that need to be interpolated.

[0094] S130, setting the extraction time, performing secondary extraction on the adjacent frames that need to be inserted, taking the adjacent frames that need to be inserted as the time center, extending the same extraction time to both sides, extracting all video frames covered by the time, and storing them as an extraction group.

[0095] In this step, the video file is first read and every few frames are extracted according to the set extraction frequency to reduce the computational burden. The extracted frames are stored in memory in the form of an array for subsequent processing. Next, each frame is processed and an image processing algorithm is applied for object detection to identify all objects in the scene. This step includes feature extraction, which extracts the features of each object (such as shape, color, outline, etc.), and object classification, which classifies the identified objects into categories (such as furniture, people, background, etc.) and records the location information.

[0096] Combine several adjacent frames to determine the temporal motion coherence of the object. The specific steps include applying the optical flow method to calculate the motion vector of the object between adjacent frames and determine the direction and speed of the object's motion. If there is an obvious change in the position of the object between two adjacent frames, but the optical flow detection shows that the object does not move naturally, the two adjacent frames are defined as adjacent frames that need to be interpolated. In addition, set the extraction duration. For each pair of adjacent frames that need to be interpolated, set an extraction duration. Take the adjacent frame as the time center and extend the same extraction duration to both sides. Extract the corresponding number of frames according to the video frame rate to form an extraction group. Finally, store all frames in the extraction group into an independent structure for subsequent lighting and shadow analysis.

[0097] By setting the extraction frequency and extraction duration, the amount of data that needs to be processed is effectively reduced, thereby reducing the computational burden and making the interpolation process more efficient. At the same time, the use of optical flow method and temporal consistency analysis can accurately identify adjacent frames that need to be interpolated, ensuring the accuracy of interpolation processing in terms of motion smoothness and visual coherence. In addition, extracting frames on both sides of adjacent frames to form extraction groups enables the interpolation algorithm to obtain richer contextual information and have a comprehensive understanding of lighting changes and object motion, thereby improving the quality of the final inserted frames. The design of this step has good dynamic adaptability, and can adjust the extraction frequency and extraction duration according to different types of video content. Whether it is a static scene or a fast-moving scene, it can effectively identify and handle problems. By accurately judging whether interpolation is needed, unnecessary interpolation in scenes where objects move quickly or change suddenly can be avoided, which can reduce the visual discomfort of the audience. In addition, the overall structured storage provides a good foundation for subsequent lighting and shadow analysis, ensuring the coherence and operability of the overall algorithm process.

[0098] S200 calculates the RGB value of each pixel, analyzes brightness and color distribution, identifies the edge of the light source, and identifies the position and intensity of the light source in the scene:

[0099] In this step, each pixel of the image in each video frame is traversed to extract its RGB value and convert these RGB values ​​into corresponding brightness values. This conversion is based on the sensitivity of the human eye to different colors, ensuring that the visual effects under real lighting conditions can be accurately reflected in subsequent analysis. Next, the system generates a color histogram, which counts the number of pixels of each color and analyzes the distribution of different colors in the image, thereby providing basic data for subsequent light source identification.

[0100] By analyzing the RGB values ​​of each pixel in detail, S2 can achieve high-precision light source identification, ensuring that the true light source information can be effectively captured even in complex lighting environments. This refined analysis method can adapt to a variety of scenarios, whether it is bright sunlight or weak indoor lighting, it can be reliably identified.

[0101] On this basis, an edge detection algorithm is used to set a threshold for intensity changes to extract edge information in the image and identify areas where the intensity changes exceed the threshold. These areas are defined as potential light source locations. Through further analysis of these potential light source locations, the system can identify the point with the highest brightness value and locate it as the specific location of the light source. In addition, the average brightness value of the pixels around the light source location is calculated to determine the intensity of the light source. This process can not only identify the exact location of the light source, but also make a judgment on the type of light source (such as natural light, artificial light, etc.), providing accurate data support for the subsequent construction of the lighting model.

[0102] This method can quickly and accurately locate potential light source areas, avoiding possible misidentification in traditional methods. This efficient processing method significantly improves the response speed and accuracy of the entire system, allowing subsequent lighting model construction to be based on accurate light source information, thereby improving the consistency and realism of lighting and shadows during interpolation.

[0103] like Figure 3 As shown, the analysis of brightness and color distribution, identification of light source edges, and identification of light source positions and intensities in the scene specifically include:

[0104] S210, analyzing the video frames in the extraction group, traversing each pixel of the image in each video frame, extracting the RGB value, and converting it into a brightness value ;

[0105] ;

[0106] in Represents the red channel value of the pixel. Represents the green channel value of the pixel. Represents the blue channel value of the pixel;

[0107] S220, generating a color histogram according to the extracted RGB values, and counting the number of pixels of each color;

[0108] S230, identifying edges in the image according to an edge detection algorithm, setting an intensity change threshold, extracting edge information, identifying areas where the intensity change exceeds the threshold, and defining them as potential light source positions;

[0109] S240, identifying a point with the highest brightness value among potential light source positions as the light source position, and calculating an average brightness value of pixels around the light source position as the light source intensity;

[0110] S250: Determine the light source type based on the light source intensity and the light source position distribution.

[0111] S300, based on the position of the light source, taking into account the factors of ambient light, diffuse reflection and specular reflection, constructs a lighting model, calculates the light intensity for each object, and analyzes the lighting effects on different objects;

[0112] In this step, by setting the ambient light intensity, the appropriate ambient light intensity value is set according to the light source intensity to simulate the scattered light in the scene, ensuring that each object can present a natural lighting effect under different lighting conditions and reduce the visual abruptness caused by the change of light source. In the lighting calculation, the lighting vector of each object surface is first calculated. The lighting vector is the direction from the light source to the object surface, which represents the propagation path of the light. Secondly, the mutual influence of lighting and shadows between adjacent objects is evaluated. By analyzing the changes in the position and direction of the object relative to the light source, the direction and intensity of the lighting vector are adjusted in time to ensure the natural transition of shadows and lighting effects.

[0113] In the calculation of light intensity, diffuse light intensity and specular light intensity are considered. For diffuse light intensity, the light intensity is determined by calculating the dot product of the normal and the light direction. The larger the dot product result, the more perpendicular the light is to the surface, and the brighter the object will appear. If the light is parallel to the surface (that is, the dot product is zero), the surface of the object will not receive direct light. In this way, the calculation of diffuse light can reflect the brightness changes of the object surface under different lighting conditions, enhancing the realism of the scene.

[0114] As for the intensity of specular reflection, its purpose is to simulate the highlight effect after light is reflected on the surface of an object, and to enhance the visual appeal of the object. By calculating the dot product of the reflection vector and the observer's line of sight, the intensity of the reflected light seen by the observer can be evaluated. When the angle between the observer and the reflected light is closer, the intensity of the reflected light increases, making the surface of the object appear more vivid and bright under the light. This phenomenon is crucial in many scenes, especially on shiny surfaces (such as metal or water), which can improve visual realism.

[0115] Finally, all lighting components are combined to form the total lighting intensity of the object. Ambient light is the most basic lighting in the scene, usually used to simulate the effect of scattered light to ensure a certain degree of visibility even in shadow areas. This prevents objects from disappearing in places with insufficient light and increases the sense of layering of the scene. Through this comprehensive calculation, a comprehensive grasp of the lighting of objects can be achieved, so that each object can show natural and harmonious brightness changes under different lighting conditions.

[0116] Through detailed lighting model calculation, more realistic lighting effects can be generated to enhance the realism of objects in the scene. For example, the introduction of ambient light allows objects to retain a certain brightness in the shadows, avoiding the "floating" phenomenon. In addition, the model can automatically adjust the lighting effect according to the position and intensity changes of the light source in the scene, which is particularly important for dynamic scenes. If the light source moves, the model will adjust the lighting calculation in real time to ensure the consistency of lighting with the previous and next frames to avoid the audience feeling inconsistent. In complex scenes, the mutual influence of lighting and shadows between multiple objects cannot be ignored. This step creates a more natural visual effect by evaluating the occlusion and lighting interactions between objects, so that the lighting relationship between different objects can be reasonably presented. By establishing an efficient lighting model, integrating the calculated lighting parameters and storing them in a data structure, subsequent steps (such as shadow analysis and interpolation processing) can quickly access lighting information and improve overall processing efficiency. In addition, the lighting model is not only suitable for indoor scenes, but also can cope with complex dynamic performances, outdoor environments and other application scenarios, and has strong versatility.

[0117] like Figure 4 As shown, the construction of the illumination model, calculation of the illumination intensity for each object, and analysis of the illumination effects on different objects specifically include:

[0118] S310, establish a lighting model and set the ambient light intensity value according to the light source intensity , used to simulate the scattered light in the scene and calculate the impact of ambient light on the surface of each object;

[0119] S320, for adjacent objects, consider the mutual influence of occlusion and illumination, evaluate the changes in shadow and illumination effects, and when the position and direction of the object relative to the light source change, adjust the illumination vector Adjust the direction and intensity of

[0120] S330: Output the calculated illumination model parameters and illumination intensity information to a data structure and store them.

[0121] In this step, the influence of ambient light on the surface of each object is calculated, specifically:

[0122] Calculate the illumination vector from the light source to the surface of the object, obtain the normal vector of the surface of the object, and calculate the diffuse illumination intensity:

[0123] ;

[0124] in, is the diffuse reflected light intensity, is the light source intensity, is the normal vector, , is the light vector;

[0125] in, ;

[0126] Calculate the viewing angle vector between the video image acquisition point and the object surface point, and calculate the reflection vector through the illumination vector and normal vector:

[0127] ;

[0128] Calculating specular intensity :

[0129] ;

[0130] in, is the mirror reflection coefficient, take 0.5, is the viewing angle vector;

[0131] Calculate global ambient light intensity :

[0132] .

[0133] S400, analyzing the illumination changes between adjacent frames, identifying the shadow area, and comparing it with the non-shadow area, identifying the available shadow area, calculating the shadow shape and position of each object based on the light source position, setting the shadow length and shape corresponding to each object according to the height of each object and the light source angle, and generating shadow information of the inserted frame;

[0134] In this step, the system automatically sets a brightness threshold to distinguish between shadow areas and non-shadow areas based on the brightness distribution of adjacent frames. This threshold is set based on an overall analysis of the light intensity in the scene. By traversing the brightness matrix of the current frame, all pixel areas below the brightness threshold are identified and marked as potential shadow areas. In this way, the system can effectively locate areas where shadows may exist, laying the foundation for subsequent processing.

[0135] Next, the identified potential shadow areas are subjected to connectivity analysis. The core of this process is to merge adjacent potential shadow areas to form a continuous shadow area, which can avoid recognition errors caused by fragmented shadow areas. In this process, the system will also filter shadow areas based on their shape and size, excluding shadow areas smaller than a certain area to reduce the possibility of misidentification, thereby enhancing the stability and accuracy of the system.

[0136] The system calculates the length of the shadow based on the position and type of the light source, combined with the height of the object, the height of the light source, and the direction. The specific length of the shadow is obtained by the relationship between the height of the light source and the height of the object, combined with the angle of the light source. This method not only ensures the relative realism of the shadow, but also ensures that the shadows of different objects remain consistent in the inserted frame.

[0137] Based on the direction and distance of the light source, the system generates the shadow shape of each object inserted into the frame and refines the outline of the shadow. This process involves geometric calculation and rendering of the shadow shape to ensure a natural visual transition of the shadow. At the same time, the system also adjusts the transparency and brightness of the shadow according to the light source intensity, distance, and ambient light intensity value to obtain the final shadow information. This adjustment not only enhances the realism of the shadow, but also ensures a harmonious relationship between the shadow and the object and ambient lighting.

[0138] Through automated brightness threshold setting and connectivity analysis, the need for manual intervention can be effectively reduced and the accuracy of shadow recognition can be improved. At the same time, by merging potential shadow areas, the continuity and naturalness of shadows are guaranteed, avoiding visual disharmony caused by scattered shadows. In addition, the shadow length calculation based on object height, light source position and direction ensures the true proportion of shadows in the inserted frame, making the shadow performance more in line with the laws of physics, thereby improving the credibility of the overall scene.

[0139] like Figure 5 As shown, the generation of shadow information of the inserted frame specifically includes:

[0140] S410, automatically setting a brightness threshold for distinguishing shadow areas from non-shadow areas according to brightness distribution of adjacent frames, traversing the brightness matrix of the current frame, identifying all pixel areas with brightness lower than the brightness threshold, and marking them as potential shadow areas;

[0141] S420, performing connectivity analysis on the potential shadow areas, merging adjacent potential shadow areas to form a continuous shadow area; filtering according to the shape and size of the shadow areas, excluding shadow areas smaller than a certain area, to reduce misidentification;

[0142] S430, calculating the shadow length according to the light source position and light source type judgment results, combined with the height of the object, the height and direction of the light source:

[0143] ;

[0144] in, represents the shadow length, Indicates the height of the object. Indicates the light angle;

[0145] S440, generating a shadow shape of each object inserted into the frame according to the direction and distance of the light source, and refining the shadow outline;

[0146] S450, adjusting the shadow transparency and shadow brightness according to the light source intensity, distance and ambient light intensity value to obtain final shadow information.

[0147] S500, tracking the motion trajectory of each object between adjacent frames, and taking the middle position of the object between adjacent frames as the position of the corresponding object in the inserted frame as the smooth motion trajectory of the object, and adjusting the shadow information of the inserted frame according to the lighting status of the adjacent frames to obtain the final image of the inserted frame.

[0148] In this step, the main task is to track the motion trajectory of objects between adjacent frames and apply this motion information to the generation of inserted frames. This process is crucial because it ensures that the position of the object in the inserted frame maintains a smooth transition between the previous and next frames, thereby improving the overall visual fluency and realism. First, the system needs to identify the objects that exist in the adjacent frames and record their status, including information such as the type, position, and center point of the object. With this information, the system can determine the list of objects that need to be tracked in the inserted frame. When calculating the position of the object in the inserted frame, the system takes the center position of the object in the adjacent frame and calculates the motion vector between them to obtain the direction and speed of the object's movement between adjacent frames. This vector not only provides information about the object's movement, but also provides a basis for subsequent position calculations, which helps to achieve a natural transition of objects in the inserted frame.

[0149] Then the middle position of the object in the inserted frame is calculated. The purpose of this calculation is to ensure that the performance of the object in the inserted frame is consistent with its motion trajectory in the adjacent frames by taking the middle point of the previous frame position and the current frame position as the position in the inserted frame, thereby avoiding visual abruptness and ensuring that the position of the object in the inserted frame neither deviates from the actual trajectory nor is coordinated with the actual motion process, thereby enhancing visual continuity.

[0150] In addition, this step also emphasizes the adjustment of lighting conditions. The system extracts the lighting information of adjacent frames and adjusts the shadow information in the inserted frame accordingly. This process ensures that the changes in lighting and shadows when objects move can be naturally reflected in the inserted frames, avoiding the unstable shadow phenomenon caused by inconsistent lighting, thereby enhancing the authenticity of the overall scene.

[0151] Through this precise object tracking and lighting adjustment, the inserted frames can be better integrated into the overall video stream, improving the overall quality of the picture, allowing viewers to experience a smoother and more natural visual effect when watching. Especially in complex dynamic scenes, such as dance performances or sports scenes, this step can automatically adapt to the rapid movement of objects and changing light environments, ensuring that each inserted frame is consistent with the actual scene, thereby effectively eliminating the phenomenon of objects "floating" in the inserted frames.

[0152] like Figure 6 As shown, the obtaining of the final image of the inserted frame specifically includes:

[0153] S510, extracting a list of objects that need to be tracked in the inserted frame by using the identified existing objects;

[0154] S520, for each object, calculate the position difference between the adjacent frames on both sides of the inserted frame to obtain the object motion vector:

[0155] ;

[0156] in, is the object's motion vector, Indicates the center position of the object in the frame after the insertion frame. Indicates the center position of the object in the frame before the inserted frame;

[0157] S530, calculating the middle position of the object in the inserted frame according to the position of the object in the adjacent frames:

[0158] ;

[0159] in, is the middle position of the object in the inserted frame;

[0160] S540, extracting the illumination status of adjacent frames, and verifying and adjusting the final shadow information;

[0161] S550: synthesize the intermediate positions, lighting states and shadow information of all objects in the inserted frame into one image to generate a final inserted frame image.

[0162] Figure 7 A structural block diagram of a computer video image generation system provided by an embodiment of the present invention, such as Figure 7 As shown, the system comprises:

[0163] The frame extraction and analysis module 100 is used to extract all frames from the video, determine the adjacent frames that need to be inserted, take the adjacent frames as the extraction center, and extract the video frames on both sides of the adjacent frames;

[0164] The light source identification and analysis module 200 is used to analyze the brightness and color distribution, identify the edge of the light source, and identify the position and intensity of the light source in the scene by calculating the RGB value of each pixel:

[0165] The illumination model building module 300 is used to build an illumination model based on the position of the light source, taking into account factors such as ambient light, diffuse reflection and specular reflection, calculate the illumination intensity for each object, and analyze the illumination effects on different objects;

[0166] The shadow recognition and calculation module 400 is used to analyze the illumination changes between adjacent frames, identify the shadow area, compare it with the non-shadow area, identify the available shadow area, calculate the shadow shape and position of each object based on the light source position, set the shadow length and shape corresponding to each object according to the height of each object and the light source angle, and generate the shadow information of the inserted frame;

[0167] The object tracking and interpolation frame generation module 500 is used to track the motion trajectory of each object between adjacent frames, and take the middle position of the object between adjacent frames as the position of the corresponding object in the interpolation frame as the smooth motion trajectory of the object, and adjust the shadow information of the interpolation frame according to the lighting status of the adjacent frames to obtain the final image of the interpolation frame.

[0168] like Figure 8 As shown, the frame extraction and analysis module 100 includes:

[0169] The video reading unit 110 is used to read the video file, set the extraction frequency to extract frames from the video, and store them in the form of an array;

[0170] The object motion continuity judging unit 120 is used to analyze the video frames in the array, identify the existing objects, and judge whether the object motion in the adjacent frames is temporally coherent in combination with the adjacent frames. If there is discontinuous object change between two adjacent frames, the two adjacent frames are defined as adjacent frames that need to be interpolated.

[0171] The secondary extraction time setting unit 130 is used to set the extraction time, perform secondary extraction on adjacent frames that need to be inserted, take the adjacent frames that need to be inserted as the time center, extend the same extraction time to both sides, extract all video frames covered by the time, and store them as an extraction group.

[0172] like Fig. 9As shown, the light source identification and analysis module 200 includes:

[0173] The pixel RGB value extraction unit 210 is used to analyze the video frames in the extraction group, traverse each pixel of the image in each video frame, extract the RGB value, and convert it into a brightness value;

[0174] A color histogram generating unit 220 is used to generate a color histogram according to the extracted RGB values ​​and count the number of pixels of each color;

[0175] The edge detection and intensity change analysis unit 230 is used to identify the edge in the image according to the edge detection algorithm, set the intensity change threshold, extract the edge information, and identify the area where the intensity change exceeds the threshold, which is defined as the potential light source position;

[0176] A light source position identification unit 240 is used to identify the point with the highest brightness value among the potential light source positions as the light source position, and calculate the average brightness value of the pixels around the light source position as the light source intensity;

[0177] The light source type determination unit 250 is used to determine the light source type based on the light source intensity and the light source position distribution.

[0178] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0179] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0180] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0181] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0182] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating a computer video image, characterized in that: The method comprises: Extract all frames from the video, determine the adjacent frames that need to be inserted, take the adjacent frames as the extraction center, and extract the video frames on both sides of the adjacent frames; By calculating the RGB value of each pixel, analyzing the brightness and color distribution, identifying the edge of the light source, and identifying the position and intensity of the light source in the scene: According to the position of the light source, taking into account the factors of ambient light, diffuse reflection and specular reflection, a lighting model is constructed to calculate the light intensity for each object and analyze the effects of light on different objects; Analyze the lighting changes between adjacent frames, identify the shadow area, and compare it with the non-shadow area to identify the available shadow area. Based on the light source position, calculate the shadow shape and position of each object. According to the height of each object and the light source angle, set the shadow length and shape corresponding to each object, and generate the shadow information of the inserted frame. Track the motion trajectory of each object between adjacent frames, and take the middle position of the object between adjacent frames as the position of the corresponding object in the inserted frame as the smooth motion trajectory of the object, and adjust the shadow information of the inserted frame according to the lighting status of the adjacent frames to obtain the final image of the inserted frame.

2. The method according to claim 1, characterized in that The step of determining adjacent frames that need to be inserted, taking the adjacent frames as extraction centers, and extracting video frames on both sides of the adjacent frames specifically includes: Read the video file, set the extraction frequency to extract frames from the video, and store them in the form of an array; Analyze the video frames in the array, identify the existing objects, and combine several adjacent frames to determine whether the object movements in several adjacent frames are temporally coherent. If there is discontinuous object change between two adjacent frames, the two adjacent frames are defined as adjacent frames that need to be interpolated. Set the extraction duration, perform secondary extraction on the adjacent frames that need to be inserted, take the adjacent frames that need to be inserted as the time center, extend the same extraction duration to both sides, extract all video frames covered by the duration, and store them as an extraction group.

3. The method according to claim 2, characterized in that The analyzing of brightness and color distribution, identifying the edge of the light source, and identifying the position and intensity of the light source in the scene specifically includes: Analyze the video frames in the extraction group, traverse each pixel of the image in each video frame, extract the RGB value, and convert it to a brightness value ; ； in Represents the red channel value of the pixel. Represents the green channel value of the pixel. Represents the blue channel value of the pixel; Generate a color histogram based on the extracted RGB values ​​and count the number of pixels of each color; According to the edge detection algorithm, the edges in the image are identified, the intensity change threshold is set, the edge information is extracted, and the area where the intensity change exceeds the threshold is identified, which is defined as the potential light source position; Identify the point with the highest brightness value among the potential light source positions as the light source position, and calculate the average brightness value of the pixels around the light source position as the light source intensity; Determine the light source type based on the light source intensity and light source position distribution.

4. The method according to claim 3, characterized in that The construction of the illumination model, calculation of the illumination intensity for each object, and analysis of the illumination effects on different objects specifically include: Establish a lighting model and set the ambient light intensity value according to the light source intensity , used to simulate the scattered light in the scene and calculate the impact of ambient light on the surface of each object; For adjacent objects, consider the interaction between occlusion and lighting, evaluate the changes in shadow and lighting effects, and adjust the lighting vector when the object changes its position and direction relative to the light source. Adjust the direction and intensity of The calculated illumination model parameters and illumination intensity information are output to the data structure and stored.

5. The method according to claim 4, characterized in that The calculation of the influence of ambient light on the surface of each object is as follows: Calculate the illumination vector from the light source to the surface of the object, obtain the normal vector of the surface of the object, and calculate the diffuse illumination intensity: ； in, is the diffuse reflected light intensity, is the light source intensity, is the normal vector, , is the light vector; in, ; Calculate the viewing angle vector between the video image acquisition point and the object surface point, and calculate the reflection vector through the illumination vector and normal vector. : ； Calculating specular intensity : ； in, is the mirror reflection coefficient, take 0.5, is the viewing angle vector; Calculate global ambient light intensity : 。 6. The method according to claim 4, characterized in that The generating of the shadow information of the inserted frame specifically includes: According to the brightness distribution of adjacent frames, the brightness threshold for distinguishing shadow areas from non-shadow areas is automatically set, the brightness matrix of the current frame is traversed, all pixel areas with brightness below the threshold are identified, and marked as potential shadow areas; Conduct connectivity analysis on potential shadow areas and merge adjacent potential shadow areas to form a continuous shadow area. According to the results of the light source position and light source type, the shadow length is calculated based on the height of the object, the height and direction of the light source: ； in, represents the shadow length, Indicates the height of the object. Indicates the light angle; Generate the shadow shape of each object inserted into the frame according to the direction and distance of the light source, and refine the shadow outline; According to the light intensity, distance and ambient light intensity value, the shadow transparency and shadow brightness are adjusted to obtain the final shadow information.

7. The method according to claim 6, characterized in that The method of tracking the motion trajectory of each object between adjacent frames, taking the middle position of the object between adjacent frames as the position of the corresponding object in the inserted frame as the smooth motion trajectory of the object, and adjusting the shadow information of the inserted frame according to the lighting state of the adjacent frames to obtain the final image of the inserted frame specifically includes: Using the identified existing objects, extract the list of objects that need to be tracked in the inserted frame; For each object, calculate the position difference between the object in the adjacent frames on both sides of the inserted frame to obtain the object motion vector: ； in, is the object's motion vector, Indicates the center position of the object in the frame after the insertion frame. Indicates the center position of the object in the frame before the inserted frame; Based on the position of the object in the adjacent frames, calculate its middle position in the inserted frame: ； in, is the middle position of the object in the inserted frame; Extract the lighting status of adjacent frames and verify and adjust the final shadow information; The intermediate positions, lighting states, and shadow information of all objects in the inserted frames are synthesized into one image to generate the final inserted frame image.

8. A computer video image generation system, characterized in that: The system comprises: The frame extraction and analysis module is used to extract all frames from the video, determine the adjacent frames that need to be inserted, take the adjacent frames as the extraction center, and extract the video frames on both sides of the adjacent frames; The light source identification and analysis module is used to analyze the brightness and color distribution, identify the edge of the light source, and identify the position and intensity of the light source in the scene by calculating the RGB value of each pixel: The illumination model building module is used to build an illumination model based on the position of the light source, taking into account ambient light, diffuse reflection and specular reflection factors, calculate the illumination intensity for each object, and analyze the illumination effects on different objects; Shadow recognition and calculation module, used to analyze the illumination changes between adjacent frames, identify the shadow area, compare it with the non-shadow area, identify the available shadow area, calculate the shadow shape and position of each object based on the light source position, set the shadow length and shape corresponding to each object according to the height of each object and the light source angle, and generate the shadow information of the inserted frame; The object tracking and interpolation frame generation module is used to track the motion trajectory of each object between adjacent frames, and take the middle position of the object between adjacent frames as the position of the corresponding object in the interpolation frame as the smooth motion trajectory of the object, and adjust the shadow information of the interpolation frame according to the lighting status of the adjacent frames to obtain the final image of the interpolation frame.

9. The system according to claim 8, characterized in that The frame extraction and analysis module comprises: The video reading unit is used to read the video file, set the extraction frequency to extract frames from the video, and store them in the form of an array; The object motion continuity judgment unit is used to analyze the video frames in the array, identify the existing objects, and combine several adjacent frames to judge whether the object motion in several adjacent frames is temporally coherent. If there is discontinuous object change between two adjacent frames, the two adjacent frames are defined as adjacent frames that need to be interpolated. The secondary extraction time setting unit is used to set the extraction time, perform secondary extraction on adjacent frames that need to be inserted, take the adjacent frames that need to be inserted as the time center, extend the same extraction time to both sides, extract all video frames covered by the time, and store them as an extraction group.

10. The system according to claim 9, characterized in that The light source identification and analysis module includes: A pixel RGB value extraction unit is used to analyze the video frames in the extraction group, traverse each pixel of the image in each video frame, extract the RGB value, and convert it into a brightness value; A color histogram generating unit, used to generate a color histogram according to the extracted RGB values ​​and count the number of pixels of each color; An edge detection and intensity change analysis unit, which is used to identify edges in the image based on an edge detection algorithm, set an intensity change threshold, extract edge information, and identify areas where the intensity change exceeds the threshold, which are defined as potential light source locations; A light source position identification unit, used to identify the point with the highest brightness value among the potential light source positions as the light source position, and calculate the average brightness value of the pixels around the light source position as the light source intensity; The light source type determination unit is used to determine the light source type based on the light source intensity and the light source position distribution.

Images