Image generation method and apparatus
By combining the motion vectors and optical flow of image frames to generate extrapolated frames, the problem of insufficient image quality in extrapolated frame schemes is solved, resulting in more natural and smoother image frames and improving the user experience.
Patent Information
- Application Number
- CN202510599597.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-09
- Publication Date
- 2026-11-10
AI Technical Summary
Existing extrapolation frame schemes still need improvement in the quality of the generated interpolated images, especially when dealing with geometric and non-geometric motion, making it difficult to generate natural and smooth image frames.
By combining the motion vectors and optical flow of image frames, extrapolated frames are generated, which fully considers the integrity of both geometric and non-geometric motion and improves image quality.
It generates more natural and smoother image frames, improves the image quality of the extrapolation frame scheme, reduces image jitter and distortion, and enhances the user experience.
Smart Images

Figure CN123071518A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image technology, and more particularly to image generation methods and apparatus. Background Technology
[0002] With the continuous innovation of real-time game rendering technology, the image quality, resolution levels, and frame rate standards of games have all shown an exponential upward trend, leading to a continuous increase in the rendering pressure on graphics processing units (GPUs). Against this backdrop, frame generation technology has emerged as a key solution to reduce rendering load.
[0003] Frame generation schemes, as a key strategy to reduce rendering load, enable the GPU to render at a low frame rate and combine this with efficient frame generation techniques to increase the frame rate, thereby achieving image quality and smoothness comparable to native high frame rate rendering. From a performance and power consumption perspective, the time and power overhead of inserting a frame is usually far less than the overhead of rendering a frame, thus significantly improving performance. Currently, mainstream frame generation techniques are mainly divided into two types: frame interpolation and frame extrapolation.
[0004] Interpolation frame schemes rely on the preceding and following frames to generate intermediate frames, and generating intermediate frames requires waiting for the arrival of the following frame, which inevitably causes delays and affects the user experience.
[0005] In contrast, the extrapolation frame scheme uses past frames to predict future frames, which theoretically can achieve zero-latency future frame generation.
[0006] However, existing extrapolation frame schemes still need improvement in the quality of the interpolated images generated. Summary of the Invention
[0007] This application provides an image generation method and apparatus, relating to the field of image technology, for improving the image quality of image frames generated by an extrapolation frame scheme. To achieve the above objective, this application adopts the following technical solution:
[0008] In a first aspect, embodiments of this application provide an image generation method, the method comprising: acquiring a first motion vector of a first image frame to be generated; acquiring the optical flow of the first image frame; and generating the first image frame based on the first motion vector, the optical flow, and a second image frame. The second image frame is the preceding image frame of the first image frame.
[0009] As can be seen, the solution provided in this application not only uses the motion vector of the image frame during the image frame generation process, but also combines the optical flow of the image frame. The motion vector can characterize the motion trend of geometric objects in the image frame, while the optical flow can characterize the motion trend of non-geometric objects (such as light and shadow and special effects) in the image frame. By fusing motion vector and optical flow, this solution can fully take into account the integrity of geometric motion and non-geometric motion when generating extrapolation frames, thereby generating more natural and smooth image frames, thus improving the image quality of the image frames generated by the extrapolation frame solution.
[0010] In one possible implementation, the acquisition of the optical flow of the first image frame specifically includes: acquiring the latent state of the second image frame; and determining the optical flow based on the latent state of the second image frame and the first motion vector.
[0011] Understandably, the latent state of the second image frame can provide the visual features of that frame (such as object outlines and textures) and the motion trends of historical frames (such as acceleration and special effects changes), while the first motion vector can characterize the linear motion trajectory of geometric objects within the first image frame. This scheme, by combining the latent state of the second image frame and the first motion vector of the first image frame, can generate the optical flow of the first image frame relatively accurately.
[0012] In one possible implementation, obtaining the latent state of the second image frame specifically includes: determining the latent state of the second image frame based on the second motion vector of the second image frame and the latent state of the third image frame.
[0013] The third image frame is the image frame preceding the second image frame.
[0014] It is understandable that the latent state of the third image frame can provide the visual features of that frame and the motion trends of historical frames, while the second motion vector can characterize the linear motion trajectory of geometric objects within the second image frame. This scheme can generate the latent state of the second image frame relatively accurately by combining the latent state of the third image frame and the second motion vector of the second image frame.
[0015] In one possible implementation, the method further includes: determining the third depth information of the first image frame based on the first depth information of the second image frame and the second depth information of the third image frame.
[0016] In one possible implementation, generating the first image frame based on the first motion vector, the optical flow, and the second image frame specifically includes generating the first image frame based on the third depth information, the first motion vector, the optical flow, and the second image frame.
[0017] As can be seen, in generating the third depth information of the first image frame, this application's solution not only uses the first depth information of the second image frame but also combines the second depth information of the third image frame. The first depth information represents the distance of each pixel in the second image frame to the camera, providing the spatial position of objects in the image, while the second depth information reflects the distance of each pixel in the previous image frame to the camera. By combining these two, the depth change trend of pixels in the time dimension can be effectively inferred. This solution can generate the third depth information of the first image frame relatively accurately by combining the first and second depth information, and then use this third depth information to generate a relatively accurate first image frame.
[0018] In one possible implementation, generating the first image frame based on the third depth information, the first motion vector, the optical flow, and the second image frame specifically includes: obtaining the weight of the first image frame; determining the optical flow processing result based on the first motion vector, the optical flow, and the weight; and generating the first image frame based on the third depth information, the optical flow processing result, and the second image frame.
[0019] In one possible implementation, obtaining the first motion vector specifically includes: determining the motion vector at a second time-domain position before the first time-domain position of the first image frame in the time domain as the first motion vector.
[0020] Understandably, motion vectors describe the distance and direction of movement of geometric pixels or objects in an image, and are crucial parameters for generating extrapolated frames. However, due to the nonlinear motion of geometric objects (such as acceleration, deceleration, or turning), directly generating extrapolated frames based on linear extrapolation of motion vectors may lead to overly aggressive predictions, resulting in image quality issues (such as image jitter or distortion). This scheme determines the motion vector of the extrapolated frame at a second time-domain position before the first time-domain position in the extrapolated frame, thereby effectively attenuating the value of the extrapolated frame's motion vector. This limits the extrapolation amplitude, ensures the temporal smoothness of image frames, and generates more stable and natural image frames.
[0021] Secondly, embodiments of this application provide an image generation apparatus. The apparatus may be an electronic device, a module applied to an electronic device (such as a processor, chip, or chip system), or a logic node, logic module, or software capable of implementing all or part of the functions of an electronic device. The apparatus includes an acquisition unit and a generation unit.
[0022] The acquisition unit is used to acquire the first motion vector of the first image frame to be generated.
[0023] The acquisition unit is also used to acquire the optical flow of the first image frame mentioned above.
[0024] The generation unit is configured to generate the first image frame based on the first motion vector, the optical flow, and the second image frame, wherein the second image frame is the image frame preceding the first image frame.
[0025] In one possible implementation, the acquisition unit is specifically used to: acquire the latent state of the second image frame; and determine the optical flow based on the latent state of the second image frame and the first motion vector.
[0026] In one possible implementation, the acquisition unit is specifically used to: determine the latent state of the second image frame based on the second motion vector of the second image frame and the latent state of the third image frame, wherein the third image frame is the previous image frame of the second image frame.
[0027] In one possible implementation, the acquisition unit is further configured to: determine the third depth information of the first image frame based on the first depth information of the second image frame and the second depth information of the third image frame;
[0028] In one possible implementation, the generation unit is specifically used to generate the first image frame based on the third depth information, the first motion vector, the optical flow, and the second image frame.
[0029] In one possible implementation, the generation unit is specifically used to: obtain the weight of the first image frame; determine the optical flow processing result based on the first motion vector, the optical flow, and the weight; and generate the first image frame based on the third depth information, the optical flow processing result, and the second image frame.
[0030] In one possible implementation, the acquisition unit is specifically used to: determine the motion vector at a second time-domain position before the first time-domain position of the first image frame in the time domain as the first motion vector.
[0031] Thirdly, embodiments of this application also provide an image generation apparatus, which includes: at least one processor, which, when the at least one processor executes program code or instructions, implements the method in the first aspect or any possible implementation thereof.
[0032] Optionally, the device may further include at least one memory for storing the program code or instructions.
[0033] In one possible implementation, the image generation device is a chip or chip system.
[0034] Fourthly, embodiments of this application also provide a chip, including: an input interface, an output interface, and at least one processor. Optionally, the chip further includes a memory. The at least one processor is used to execute code in the memory, and when the at least one processor executes the code, the chip implements the method in the first aspect or any possible implementation thereof.
[0035] Alternatively, the chip described above can also be an integrated circuit.
[0036] Fifthly, embodiments of this application also provide a computer-readable storage medium for storing a computer program, the computer program including methods for implementing the first aspect above or any possible implementation thereof.
[0037] In one possible implementation, the computer-readable storage medium is a non-transitory computer-readable medium.
[0038] Sixthly, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to implement the method described in the first aspect or any possible implementation thereof.
[0039] The image generation apparatus, computer-readable storage medium, computer program product, and chip provided in this embodiment are all used to execute the image generation method provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the image generation method provided above, and will not be repeated here. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0041] Figure 1 This is a schematic diagram of interpolated frames;
[0042] Figure 2 This is a schematic diagram of an extrapolated frame;
[0043] Figure 3 A flowchart illustrating an image generation method provided in an embodiment of this application;
[0044] Figure 4 A schematic diagram illustrating the process of generating the first motion vector provided in an embodiment of this application;
[0045] Figure 5 A schematic diagram illustrating the time-domain location provided in an embodiment of this application;
[0046] Figure 6 A schematic diagram of motion vector attenuation provided in an embodiment of this application;
[0047] Figure 7 A schematic diagram of a hole mask.
[0048] Figure 8 A schematic diagram of the first network model provided in an embodiment of this application;
[0049] Figure 9 A schematic diagram of the second network model provided in the embodiments of this application;
[0050] Figure 10 A schematic diagram of the depth information difference provided in the embodiments of this application.
[0051] Figure 11 A schematic diagram of the third network model provided in the embodiments of this application;
[0052] Figure 12 A schematic diagram of the first network model provided in an embodiment of this application;
[0053] Figure 13 A flowchart illustrating another image generation method provided in an embodiment of this application;
[0054] Figure 14 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application;
[0055] Figure 15 This is a schematic diagram of the structure of a chip provided in an embodiment of this application;
[0056] Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the embodiments of this application.
[0058] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.
[0059] The terms "first" and "second," etc., in the specification and drawings of the embodiments of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.
[0060] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.
[0061] It should be noted that in the description of the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.
[0062] For ease of understanding, the following explanations are provided for the terminology used in this plan:
[0063] Motion vector (MV): Represents the direction and velocity of a pixel or object's motion between two frames, typically stored as a two-dimensional vector. During rasterization, motion vectors are calculated through vertex transformation or geometric interpolation and used in post-processing techniques such as temporal anti-aliasing and motion blur synthesis. Motion vectors can also assist in dynamic mask generation, improving the rendering accuracy and efficiency of dynamic objects.
[0064] Optical flow is pixel-level motion information estimated through image processing techniques. It can capture the movement of all pixels in an image, including geometric and non-geometric effects.
[0065] Depth information refers to the vertical distance of each pixel in the scene from the camera, typically stored as a floating-point value in a depth buffer. Depth values are used to determine occlusion relationships between objects and to filter out the final visible pixels through a depth test. In the rendering pipeline, depth information is derived from the geometric data after vertex processing during the rasterization stage and is a key input for per-patch meta-operations.
[0066] A dynamic mask is a runtime-generated binary or multi-channel mask used to identify dynamic elements in a scene that require special processing (such as moving objects or special effects areas). For example, in layered rendering, masks can be used to distinguish different layers and control the camera to render only the content of a specified layer. Dynamic masks can also be used in post-processing to isolate specific areas (such as motion blur or local color correction).
[0067] Forward warping is a technique in computer graphics used for image deformation or synthesis. Its core principle is to directly map pixels from the source image to corresponding positions in the target image to achieve the transformation. While intuitive and easy to implement, this method, by not fully considering the dynamic deformation and displacement of pixels during the mapping process, can lead to problems such as pixel overlap, holes, or jagged edges in the target image, thus degrading image quality. In contrast, backward warping calculates the optimal matching points of the target pixels in the source image in reverse, typically preserving details more accurately and generating high-quality deformation results.
[0068] Backward warping is a computer vision technique used for image deformation and registration. Its core principle is to reverse-engineer the corresponding pixels in the source image from the target image to determine how to accurately map the content of the source image to the target location. This method calculates the optimal matching point for each target pixel in the source image, avoiding pixel overlap or holes, thus generating high-precision deformation results. Its advantage lies in effectively preserving image details and continuity, and it is widely used in image synthesis, video stabilization, and interpolation to achieve smooth and natural visual effects. Unlike the direct mapping of forward projection, backward warping significantly improves the quality and accuracy of image transformation through the inverse process.
[0069] Convolutional layer: The core component of convolutional neural network. It performs local weighted operations by sliding convolutional kernels (filters) on the input data, extracting feature maps layer by layer from low-level features such as edges and lines to high-level abstract features such as shapes and objects. This computation method based on local receptive fields not only preserves spatial correlation, but also greatly reduces the number of model parameters through parameter sharing, significantly improving computational efficiency and the hierarchical expressive ability of feature extraction.
[0070] Long Short-Term Memory (LSTM) networks are a special type of recurrent neural network specifically designed to address the "vanishing gradient" or "exploding gradient" problems that traditional recurrent neural networks often encounter when processing long sequences of data. Its core lies in the introduction of a "gating mechanism" and "cell states." Through three key structures—forget gate, input gate, and output gate—LSTM can autonomously decide which information needs to be retained long-term, which needs to be forgotten, and which features in the current input are worth paying attention to. The cell state, acting as a "memory channel" throughout the network, allows information to be stably transmitted across multiple time steps, effectively capturing long-distance dependencies. This characteristic makes LSTM perform exceptionally well in scenarios requiring the processing of sequential data, such as natural language processing (e.g., machine translation, text generation) and time series prediction (e.g., weather forecasting), and it excels particularly in handling complex tasks with significant delays between input and output.
[0071] A convolutional long short-term memory (LSTM) layer is a neural network layer that combines the architectures of convolutional neural networks (CNNs) and long short-term memory (LSTM) networks. This type of layer is particularly suitable for processing sequential data, especially data with spatial and temporal dimensions, such as video frame sequences, meteorological data, and time-series images. In traditional LSTMs, the input data is processed as a one-dimensional sequence, while convolutional LSTMs, by introducing convolutional operations, enable the network to process both two-dimensional spatial and temporal information simultaneously. Convolutional operations allow the network to learn spatial features within local regions, while LSTM handles temporal dependencies. This combination allows convolutional LSTMs to capture more complex and abstract features when processing spatially structured time-series data.
[0072] U-Net is a convolutional neural network specifically designed for image segmentation. Its core lies in its unique encoder-decoder symmetric structure and skip connections. The encoder progressively extracts high-dimensional semantic features from the image through downsampling, while the decoder recovers target details and spatial location information through upsampling. Skip connections directly pass feature maps from each layer of the encoder to the corresponding layer of the decoder, achieving multi-scale feature fusion. This preserves local details while utilizing global contextual information. This structure makes UNet stand out in scenarios with small sample data, such as medical image segmentation, and it excels at pixel-level precise localization of complex targets in images, while avoiding the spatial information loss caused by downsampling.
[0073] Frame generation schemes are a key strategy for reducing rendering load by having the GPU render at a low frame rate. Currently, the mainstream frame generation techniques are mainly divided into two types: interpolated frames and extrapolated frames.
[0074] Figure 1 A schematic diagram of the interpolation frame scheme is shown. For example... Figure 1 As shown, the interpolation frame scheme generates intermediate frames by using the preceding and following frames. This scheme relies on future frames; for example, to generate and display frame t+0.5, it must wait for frame t+1 to finish rendering, thus introducing latency issues. For example, increasing the frame rate from 30 frames per second (FPS) to 60 FPS will introduce approximately 20 milliseconds of latency. Increased latency in system input response leads to inconsistencies between user actions and actual feedback, resulting in responsiveness issues and dizziness, impacting the user experience. Furthermore, when the interpolation scheme is activated, the frames rendered by the engine are not directly displayed; this requires not only an additional buffer for storage but also modifications to the display strategy.
[0075] Figure 2 A schematic diagram of the extrapolation frame scheme is shown. For example... Figure 2 As shown, the extrapolation frame scheme relies only on historical frames and not on future frames, generating new frames through extrapolation. For example, to generate frame t+1.5, only frames up to t+1.5 need to be rendered, without waiting for frame t+2 to be rendered. Therefore, it does not introduce latency issues and theoretically can achieve zero-latency future frame generation, making it more suitable for real-time rendering applications. However, the quality of the interpolated images generated by existing extrapolation frame schemes still needs improvement.
[0076] To this end, embodiments of this application provide an image generation method and apparatus for improving the image quality of image frames generated by an extrapolation frame scheme.
[0077] The technical solutions provided in the embodiments of this application can be applied to electronic devices.
[0078] For example, this solution can be applied to various terminal electronic devices, such as mobile phones, personal computers (PCs), tablets, wearable devices, in-vehicle devices, augmented reality (AR) and virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), game consoles, smart TVs, digital cameras, smartwatches / bracelets, industrial control terminals, medical imaging equipment, drone flight control systems, robot vision processing units, foldable screen devices, holographic projection devices, and other electronic devices.
[0079] For example, this solution can also be applied to electronic devices such as servers, cloud servers, cloud computing platforms, edge computing nodes, distributed server clusters, and multi-access edge computing devices.
[0080] Figure 3 This application illustrates an image generation method provided by an embodiment of the present application. This method can be executed by an electronic device with frame generation capabilities, such as... Figure 3 As shown, the method includes:
[0081] S301. Obtain the first motion vector of the first image frame to be generated.
[0082] In one possible implementation, obtaining the first motion vector specifically includes:
[0083] Obtain the second motion vector of the second image frame.
[0084] The first motion vector is determined based on the second motion vector of the second image frame.
[0085] The second image frame is the image frame preceding the first image frame.
[0086] For example, if the first image frame is frame t+0.5, the second image frame is frame t, and the third image frame is frame t-1, the first image frame is an extrapolated frame, and the second and third image frames are rendered frames, then the second motion vector (f) (t-1)→(t)Let f represent the distance each pixel in the third image frame (t-1 frame) moves from the second image frame (t frame). The first and second image frames are spaced 0.5 frames apart. After obtaining the second motion vector, extend the straight line of the second motion vector by half (if the interval is 0.75 frames, extend it by 0.75 times). This gives the position of each pixel in the second image frame (t frame) in the first image frame (t+0.5 frame). Then, calculate the distance each pixel moves from the second image frame (t frame) to the first image frame (t+0.5 frame) to obtain the first motion vector (f). (t)→(t+0.5) ).
[0087] like Figure 4 As shown, if the first image frame is t+0.5, the second image frame is t, and the third image frame is t-1, the second motion vector of the first pixel in the second image frame (t frame) is the distance the pixel moves from the third image frame (t-1 frame) to the second image frame (t frame). The interval between the third image frame (t-1 frame) and the second image frame (t frame) is 1 frame, and the interval between the second image frame (t frame) and the first image frame (t+0.5 frame) is 0.5 frames. Therefore, the distance the first pixel moves from the second image frame (t frame) to the first image frame (t+0.5 frame) is half the distance it moves from the third image frame (t-1 frame) to the second image frame (t frame). By linearly extrapolating the motion vector of the first pixel by half, its position in the first image frame (t+0.5 frame) can be determined. Thus, by calculating the distance the first pixel moves from the second image frame (t frame) to the first image frame (t+0.5 frame), the first motion vector of the pixel can be obtained.
[0088] For example, after obtaining the second motion vector of the second image frame, the first motion vector is obtained based on the forward warping of the second image frame. That is, the second motion vector is multiplied by a coefficient value (0.5 in the case of double interpolation, 1 / 3 in the case of triple interpolation, and 0.25 in the case of quadruple interpolation) to obtain the first motion vector.
[0089] In one possible implementation, obtaining the second motion vector specifically includes:
[0090] Acquire the second and third image frames.
[0091] The second motion vector is determined based on the second and third image frames.
[0092] The third image frame is the image frame preceding the second image frame.
[0093] For example, if the first image frame is t+0.5, the second image frame is t, and the third image frame is t-1, after obtaining the second image frame (t) and the third image frame (t-1), preprocess the second image frame (t) and the third image frame (t-1), and then calculate the distance each pixel in the third image frame (t-1) moves from the second image frame (t) to the second image frame (t), the second motion vector can be obtained.
[0094] Optionally, the preprocessing described above may include at least one of foreground dilation, stroke correction, or region filtering.
[0095] Foreground dilation refers to expanding the mask of foreground objects in an image frame to protect the boundary region.
[0096] Edge correction refers to smoothing the foreground edges in an image frame to reduce discontinuities in motion vectors.
[0097] Region filtering refers to filtering out regions in an image frame where motion vectors are meaningless (such as areas with light and shadow, special effects, and particle distribution).
[0098] In another possible implementation, obtaining the first motion vector specifically includes:
[0099] The motion vector at the second time-domain position before the first time-domain position of the first image frame is determined as the first motion vector.
[0100] For example, if the second motion vector is greater than the first threshold, the motion vector at the second time domain position before the first time domain position of the first image frame can be determined as the first motion vector.
[0101] like Figure 5 As shown, if the first image frame is frame t+0.5, the second image frame is frame t, and the fourth image frame is frame t+1, the first image frame (frame t+0.5) is located at the first time domain position, the second image frame (frame t) is located at the third time domain position, and the fourth image frame (frame t+1) is located at the fourth time domain position. The first time domain position is located between the third and fourth time domain positions. When the second motion vector is greater than 5 pixels, the motion vector at the second time domain position before the first time domain position of the first image frame can be determined as the first motion vector.
[0102] For example, if the original frame rate is 30 FPS, the frame rate can be increased to 60 FPS using frame interpolation. The first image frame is t+0.5, the second image frame is t, and the third image frame is t-1. The black ball moves quickly to the right at a speed of 20 pixels / frame before the second image frame (t), but slows down between the second image frame (t) and the fourth image frame (t+1), changing its movement to the right at a speed of 8 pixels / frame.
[0103] like Figure 6 As shown, if the first motion vector is not attenuated, i.e., the motion vector of the first temporal position of the first image frame is directly used as the first motion vector, the movement distance from the second image frame (frame t) to the first image frame (frame t+0.5) is 10 pixels (20 pixels / frame × 0.5 frames). However, the movement distance from the second image frame (frame t) to the fourth image frame (frame t+1) is only 8 pixels (8 pixels / frame × 1 frame). This will cause the predicted position of the black ball in the first image frame (frame t+0.5) to exceed the actual position in the fourth image frame (frame t+1), thus causing image jitter and unnatural phenomena.
[0104] like Figure 6 As shown, if the first motion vector is attenuated, that is, if the motion vector at the second time domain position (e.g., the time domain position of frame t+0.3) before the first time domain position is used as the first motion vector, then the movement distance from the second image frame (frame t) to the first image frame (frame t+0.5) is 6 pixels (20 pixels / frame × 0.3 frames), which is less than the movement distance from the second image frame (frame t) to the fourth image frame (frame t+1). This avoids the predicted position of the black ball in the first image frame (frame t+0.5) exceeding the actual position in the fourth image frame (frame t+1), making the motion trajectory of objects in the image smoother and more natural, effectively reducing image jitter.
[0105] Understandably, motion vectors describe the distance and direction of movement of geometric pixels or objects in an image, and are crucial parameters for generating extrapolated frames. However, due to the nonlinear motion of geometric objects (e.g., acceleration, deceleration, or turning), directly generating extrapolated frames based on linear extrapolation of motion vectors may lead to overly aggressive predictions, resulting in image quality issues (e.g., image jitter or distortion). This scheme effectively attenuates the value of the extrapolated frame's motion vector by determining the motion vector at a second time-domain position before the first time-domain position of the extrapolated frame. This limits the extrapolation amplitude, ensures the temporal smoothness of image frames, and generates more stable and natural image frames.
[0106] Optionally, the second time-domain position B can satisfy:
[0107] C + 0.6 × D ≤ B < A
[0108] Where A is the first time domain position, C is the third time domain position, and D is the time interval between the first time domain position and the third time domain position.
[0109] For example, if a frame (t+0.5) is inserted at the midpoint (0.5 frames) between frame t and frame t+1, a double superframe is formed. In this case, the time difference D between the first and third time domain positions is 0.5 frames. If the second time domain position is 1 frame, then the time domain interval where the second time domain position is located is [1+(0.6×0.5) frames, 1+0.5 frames), that is, [1.3 frames, 1.5 frames].
[0110] For example, if a frame (t+0.5) is inserted at the midpoint (0.5 frames) between frame t and frame t+1, a double superframe is formed. In this case, the time difference D between the first and third time domain positions is 0.5 frames. If the second time domain position is 5 frames, then the time domain interval where the second time domain position is located is [5+(0.6×0.5) frames, 5+0.5 frames), that is, [5.3 frames, 5.5 frames].
[0111] In one possible implementation, obtaining the first motion vector of the first image frame to be generated specifically includes:
[0112] The second image frame is then rendered.
[0113] For example, a second image frame can be obtained using the main rendering pipeline and temporal anti-aliasing (TAA).
[0114] The main rendering pipeline refers to the core process of converting a 3D scene into a 2D screen image. It is the foundation of real-time graphics rendering (such as games and interactive applications) and typically includes input assembly, vertex processing, clipping and projection, rasterization, pixel processing, testing and blending, and output to the framebuffer.
[0115] TAA is an anti-aliasing technique that utilizes the time dimension (multi-frame data) to reduce jagged edges and noise by mixing the current frame with historical frames. It is often used in real-time rendering.
[0116] Obtain the first motion vector of the first image frame to be generated.
[0117] S302, Obtain the optical flow of the first image frame.
[0118] In one possible implementation, the acquisition of the optical flow of the first image frame specifically includes:
[0119] Obtain the latent state of the second image frame.
[0120] The optical flow of the first image frame is determined based on the latent state of the second image frame and the first motion vector.
[0121] For example, after obtaining the latent state of the second image frame, the optical flow of the first image frame can be determined based on the latent state of the second image frame, the first motion vector, the color information of the second image frame, and the hole mask.
[0122] The latent state of the second image frame is used to express higher-order motion trends (i.e., motion trends between multiple frames), which can be used to correct linear trends (such as optical flow and motion vectors).
[0123] The color information of the second image frame can be red-green-blue (RGB) data, red-green-blue alpha (RGBA) data, or luminance and chrominance (YUV) data of the second image frame.
[0124] A hole mask is used to represent the background area exposed after an object (or pixel) moves between frames (i.e., the hole left after the foreground object moves away). For example... Figure 7 As shown, between frame t and frame t+0.5, a black sphere moves to the right. In frame t+0.5, the black sphere, which was originally in position in frame t, reveals a gray spherical background area due to its movement. This gray spherical area is the corresponding hole mask, marking the background area (gray area) exposed due to the movement of the foreground object (black sphere).
[0125] Specifically, such as Figure 8 As shown, after obtaining the latent state of the second image frame, the latent state of the second image frame, the first motion vector, the color information of the second image frame, and the hole mask can be input into the first network model to obtain the optical flow of the first image frame output by the first network model.
[0126] Optionally, the first network model can be a UNet model, a gated recurrent unit (GRU), a Transformer model, or a pyramid warping and cost volume network (PWCNet).
[0127] Understandably, the latent state of the second image frame can provide the visual features of that frame (such as object outlines and textures) and the motion trends of historical frames (such as acceleration and special effects changes), while the first motion vector can characterize the linear motion trajectory of geometric objects within the first image frame. This scheme, by combining the latent state of the second image frame and the first motion vector of the first image frame, can generate the optical flow of the first image frame relatively accurately.
[0128] In another possible implementation, the acquisition of the optical flow of the first image frame specifically includes:
[0129] The optical flow estimation algorithm is used to determine the optical flow of the first image frame based on the second image frame.
[0130] For example, the optical flow of the first image frame is estimated using an optical flow estimation algorithm based on the pixel brightness changes of the second and third image frames.
[0131] Alternatively, the optical flow estimation algorithm described above can be the Lucas-Kanade algorithm or the Horn-Schunck algorithm.
[0132] The Lucas-Kanade algorithm calculates optical flow by assuming uniform motion in local regions and using the brightness invariance assumption.
[0133] The Horn-Schunck algorithm estimates optical flow by optimizing the energy function, assuming a globally smooth optical flow field.
[0134] In one possible implementation, obtaining the latent state of the second image frame specifically includes:
[0135] The latent state of the second image frame is determined based on the second motion vector of the second image frame and the latent state of the third image frame.
[0136] For example, after obtaining the latent state of the third image frame, the latent state of the second image frame can be determined based on the latent state of the third image frame, the second motion vector, and the color information of the second image frame.
[0137] Specifically, such as Figure 9 As shown, after obtaining the latent state of the third image frame, the latent state of the third image frame, the second motion vector, and the color information of the second image frame can be input into the second network model to obtain the optical flow of the first image frame output by the second network model.
[0138] Alternatively, the second network model can be an LSTM model, a GRU model, or a Transformer model.
[0139] For example, the second network model can be a multi-layer neural network model that uses one Convolutional Layer and one Convolutional LSTM Layer.
[0140] It is understandable that the latent state of the third image frame can provide the visual features of that frame and the motion trends of historical frames, while the second motion vector can characterize the linear motion trajectory of geometric objects within the second image frame. This scheme can generate the latent state of the second image frame relatively accurately by combining the latent state of the third image frame and the second motion vector of the second image frame.
[0141] S303. Generate the first image frame based on the first motion vector, the optical flow of the first image frame, and the second image frame.
[0142] For example, the second image frame can be back-sampled based on the first motion vector and the optical flow of the first image frame to generate the first image frame.
[0143] In one possible implementation, the method further includes: determining the third depth information of the first image frame based on the first depth information of the second image frame and the second depth information of the third image frame.
[0144] For example, the depth information difference between the first depth information of the second image frame and the second depth information of the third image frame can be calculated based on the first depth information of the second image frame and the second depth information of the third image frame, and then the third depth information of the first image frame can be determined based on the depth information difference and the first depth information.
[0145] For example, the first depth information of the second image frame is 0.8 and the second depth information is 1. The depth information difference is calculated to be -0.2 (0.8-1) based on the first depth information and the second depth information being 1. The third depth information of the first image frame is determined to be 0.7 (0.8+0.5×-0.2) based on the depth information difference and the first depth information.
[0146] Optionally, the third depth information can satisfy:
[0147] depth_3=W1×depth_1+W2×(depth_1-depth_2)
[0148] Where depth_3 is the third depth information, depth_1 is the first depth information, depth_2 is the second depth information, W1 is the first weight, and W2 is the second weight.
[0149] The value of W1 can be 1 or other values (such as 0.9, 0.95, 1.05, 1.1, etc.).
[0150] The value of W2 can be related to the distance between the first image frame and the second image frame.
[0151] For example, if the first image frame is t+0.5 frames and the second image frame is t frames, and the first image frame (t+0.5 frames) and the second image frame (t frames) are 0.5 frames apart, then the value of W2 can be 0.5.
[0152] For example, if the first image frame is t+0.75 frames and the second image frame is t frames, and the first image frame (t+0.75 frames) and the second image frame (t frames) are 0.75 frames apart, then the value of W2 can be 0.75.
[0153] As can be seen, in generating the third depth information of the first image frame, this application's solution not only uses the first depth information of the second image frame but also combines the second depth information of the third image frame. The first depth information represents the distance of each pixel in the second image frame to the camera, providing the spatial position of objects in the image, while the second depth information reflects the distance of each pixel in the previous image frame to the camera. By combining these two, the depth change trend of pixels in the time dimension can be effectively inferred. This solution can generate the third depth information of the first image frame relatively accurately by combining the first and second depth information, and then use this third depth information to generate a relatively accurate first image frame.
[0154] like Figure 10 As shown, during the period from frame t to frame t+0.5, the white ball moves downwards, and the distance between it and the black ball gradually decreases. By frame t+0.5, the white ball and the black ball partially overlap. The depth information of the black ball in both frames t-1 and t is 0.75, while the depth information of the white ball in frame t-1 is 1, and the depth information in frame t is 0.8.
[0155] If the depth information difference is not used, and the depth information (i.e., the second depth information) of the black ball and the white ball in frame t is directly used as their depth information in frame t+0.5, then the depth information of the black ball in frame t+0.5 is 0.75, and the depth information of the white ball in frame t+0.5 is 0.8. Since the depth information of the black ball is less than that of the white ball in frame t+0.5, the overlapping part should display the pixels of the black ball.
[0156] However, if the depth information difference and the depth information of frame t are used, then:
[0157] The depth information difference between the black ball in frame t-1 and frame t is 0.8 - 0.8 = 0, and the depth information difference between the white ball in frame t-1 and frame t is 0.8 - 1 = -0.2.
[0158] The depth information of the black ball at frame t+0.5 is 0.75 + 0.5 × 0 = 0.75, and the depth information of the white ball at frame t+0.5 is 0.8 + 0.5 × -0.2 = 0.7.
[0159] Since the depth information of the white ball is less than that of the black ball in frame t+0.5, the overlapping area should display white ball pixels instead of black ball pixels.
[0160] pass Figure 10 It can be seen that using depth information difference can more accurately predict the depth information of an object in the first image frame, thereby avoiding errors caused by relying solely on the depth information of the second image frame. In the embodiments of this application, by using depth information difference, the accuracy of depth information prediction is effectively improved, making the generated image frame closer to the real scene and improving the quality and accuracy of the interpolated frame image generation.
[0161] In one possible implementation, generating the first image frame based on the third depth information, the first motion vector, the optical flow, and the second image frame specifically includes:
[0162] Obtain the weights of the first image frame.
[0163] The weight of the first image frame can also be called the weight map of the first image frame or the fused weight map of the first image frame.
[0164] For example, the weight of the first image frame is determined based on the latent state of the second image frame, the first motion vector, the color information of the second image frame, and the hole mask.
[0165] Specifically, such as Figure 11 As shown, the latent state of the second image frame, the first motion vector, the color information of the second image frame, and the hole mask can be input into the third network model to obtain the weights of the first image frame output by the third network model.
[0166] Alternatively, the third network model can be a UNet model, GRU, or Transformer.
[0167] Specifically, such as Figure 12 As shown, the latent state of the second image frame, the first motion vector, the color information of the second image frame, and the hole mask can be input into the first network model to obtain the weights and optical flow of the first image frame output by the first network model.
[0168] The optical flow processing result is determined based on the first motion vector, the optical flow of the first image frame, and the weight of the first image frame.
[0169] Optionally, the optical flow processing results can satisfy:
[0170] F! = F×M + f×(1-M)
[0171] Where F! is the optical flow processing result, F is the optical flow of the first image frame, M is the weight of the first image frame, and f is the first motion vector.
[0172] For example, if the first image frame is t+0.5 frames, then the optical flow processing result can satisfy:
[0173]
[0174] in, For the optical flow processing results, F t→(t+0.5) Let f be the optical flow of the first image frame (t+0.5 frame), M be the weight of the first image frame (t+0.5 frame), and f be the weight of the first image frame (t+0.5 frame). t→(t+0.5) It is the first motion vector, that is, the motion vector of the first image frame (t+0.5 frames).
[0175] The first image frame is generated based on the third depth information, the optical flow processing result, and the second image frame.
[0176] For example, the depth of each pixel in the first image frame is determined based on the third depth information, and the second image frame is backsampled based on the optical flow processing result to generate the first image frame.
[0177] Optionally, the first image frame may satisfy:
[0178] I1 = W(I2, F!)
[0179] Where I1 is the first image frame, W is the backsampling, I2 is the second image frame, and F! is the optical flow processing result.
[0180] For example, if the first image frame is t+0.5 frames and the second image frame is t frames, then the first image frame (t+0.5 frames) can satisfy:
[0181]
[0182] Among them, I t+0.5 Let W be the first image frame (t+0.5 frames), W be the backsampling value, and I be the image frame. t This is the second image frame (frame t). This is the result of optical flow processing.
[0183] In one possible implementation, after obtaining the weights of the first image frame, the weights of the first image frame can also be filtered.
[0184] Optionally, the above filtering methods can be Gaussian filtering, mean filtering, median filtering, bilateral filtering, guided filtering, morphological filtering, or adaptive filtering.
[0185] Understandably, the first image frame can be divided into multiple image blocks (image regions), each potentially assigned a different weight. Significant differences in weight values between adjacent regions can lead to image quality issues. By filtering the weights of the first image frame, the transition between adjacent image blocks can be smoother, thus preventing image quality problems. For example, when an image block has a higher weight while its neighboring blocks have lower weights, filtering allows for a gradual transition in weight values, reducing abrupt changes and maintaining image continuity and smoothness, thereby further improving the quality of the generated image.
[0186] In one possible implementation, the method may further include: training a first network model and a second network model.
[0187] For example, a video sequence can be collected as training data, some image frames in the video sequence can be used as rendering frames, and other image frames can be considered as frames to be generated. Then, the first network model and the second network model can be jointly trained based on the training data.
[0188] For example, collected game data sequences can be used as training data. Each game data sequence includes 300 image frames, for a total of 2000 sequences, with a frame rate of 60 frames per second. Odd-numbered frames are considered rendering frames and will be used as input data, while even-numbered frames are considered frames to be generated and will only be used for supervision during training, serving as ground truth. The training data is first preprocessed (e.g., using GPU-pre) and then input into the first and second network models.
[0189] The training network (such as the adaptive moment estimation (Adam) optimizer) was used for 100 training epochs. The batch size was 8, and the initial learning rate was 1e-4. At epochs 60 and 80, the learning rate was reduced to 0.1 times.
[0190] During training, extrapolated frames are generated using data from the outputs of the first and second network models. The generated extrapolated frames are then compared with the ground truth to calculate losses (such as L2 loss, L1 loss, peak signal-to-noise ratio (PSNR), or structural similarity index (SSIM)).
[0191] GroundTruth is used to measure the model's performance. For example, by comparing extrapolated frames with the true even-numbered frames, the loss can be calculated to determine the accuracy and quality of the results generated by the first and second network models.
[0192] Batch size refers to the number of training samples the model processes each time the parameters are updated. The size of the batch size affects the stability and convergence speed of training: a small batch size may cause gradient fluctuations but helps to explore the parameter space; a large batch size provides a more stable gradient estimate. For example, a batch size of 8 means that the model processes 8 training samples each time the parameters are updated.
[0193] The learning rate is a hyperparameter that controls the step size of the model during each parameter update. The learning rate determines the magnitude of model parameter adjustments: too high a learning rate may cause the model to fail to converge (skipping the optimal solution), while too low a learning rate will make training too slow.
[0194] GPU-pre is a preprocessing operation performed on the GPU on input data (such as the odd-numbered frames mentioned above). It is used to augment the data, extract features, and accelerate the training process, thereby improving model performance and efficiency.
[0195] As can be seen from S301 to S303, the solution provided in this application not only uses the motion vector of the image frame during the generation of the image frame, but also combines the optical flow of the image frame. The motion vector can characterize the motion trend of geometric objects in the image frame, while the optical flow can characterize the motion trend of non-geometric objects (such as light and shadow and special effects) in the image frame. By fusing motion vector and optical flow, this solution can fully take into account the integrity of geometric motion and non-geometric motion when generating extrapolation frames, thereby generating more natural and smooth image frames, thus improving the image quality of the image frames generated by the extrapolation frame solution.
[0196] In addition, the solutions provided in this application do not involve modification of the graphics pipeline or impose additional requirements on the graphics pipeline, and have high compatibility and versatility, and can be widely applied in various image processing and video generation scenarios.
[0197] The following is combined Figure 13This application describes the process of generating frame t+0.5 using an embodiment of the scheme, as follows: Figure 13 As shown, the process includes:
[0198] S1301, Motion Vector Preprocessing.
[0199] For example, after acquiring the second image frame (frame t) and the third image frame (frame t-1), preprocessing such as foreground dilation, edge correction, and region filtering can be performed on the second and third image frames. Then, based on the preprocessed second and third image frames, a second motion vector (f) representing the distance each pixel in the third image frame has moved from the second image frame can be calculated. (t-1)→(t) ).
[0200] S1302, Project forward based on the displayed first-stage trend.
[0201] For example, by extending the MV line of each pixel in the second image frame (frame t) by half based on the second image frame (frame t-1) and the third image frame (frame t-1), the position of each pixel in the third image frame (frame t) in the second image frame (frame t) can be obtained, that is, based on the second motion vector (f) that represents the trend of a certain stage. (t-1)→(t) The first motion vector (f) is obtained by forward projection. (t)→(t+0.5) ).
[0202] S1303. Determine the latent state of the second image frame.
[0203] For example, the latent state of the third image frame (frame t-1), the second motion vector, and the color information of the second image frame (frame t) can be input into a neural network model with one convolutional layer and one convolutional LSTM layer to obtain the latent state of the second image frame representing the implicit higher-order trend output by the neural network model.
[0204] S1304. Determine the optical flow and weights of the first image frame based on implicit higher-order trends.
[0205] For example, the latent state of the second image frame (frame t), the color information of the second image frame, the hole mask, and the first motion vector can be input into the UNet model to obtain the weights and optical flow of the first image frame output by the UNet model.
[0206] Specifically, the optical flow can have 2 channels, the color information can have 3 channels, the latent state can have 8, 16, 32, 64, or other channels, the hole mask can have 1 channel, and the weights can have 1 channel.
[0207] S1305, reverse sampling is used to obtain the first image frame.
[0208] For example, the optical flow processing result is determined based on the first motion vector, the optical flow of the first image frame, and the weights of the first image frame:
[0209]
[0210] Then, the first image frame is obtained by backsampling based on the optical flow processing results:
[0211]
[0212] in, For the optical flow processing results, F t→(t+0.5) Let f be the optical flow of the first image frame (t+0.5 frame), M be the weight of the first image frame (t+0.5 frame), and f be the weight of the first image frame (t+0.5 frame). t→(t+0.5) Let I be the first motion vector, that is, the motion vector of the first image frame (t+0.5 frames). t+0.5 Let W be the first image frame (t+0.5 frames), W be the backsampling value, and I be the image frame. t This is the second image frame (frame t).
[0213] The image generation apparatus used to perform the above image generation method is described below.
[0214] It is understood that, in order to achieve the above-mentioned functions, the image generation apparatus includes hardware and / or software modules that perform the respective functions. Based on the algorithm steps of the examples described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of the embodiments of this application.
[0215] This application embodiment can divide the image generation device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0216] When dividing each function into modules according to its corresponding function. Figure 14The diagram illustrates a possible composition of the image generation apparatus involved in the above embodiments. The apparatus may be an electronic device, a module applied to an electronic device (such as a processor, chip, or chip system), or a logic node, logic module, or software that can realize all or part of the functions of an electronic device.
[0217] like Figure 14 As shown, the image generation apparatus 1400 may include an acquisition unit 1401 and a generation unit 1402.
[0218] The acquisition unit 1401 is used to acquire the first motion vector of the first image frame to be generated.
[0219] The acquisition unit 1401 is also used to acquire the optical flow of the first image frame mentioned above.
[0220] For example, the acquisition unit 1401 can be used to perform Figure 3 S301 and S302 in the method shown.
[0221] The generation unit 1402 is used to generate the first image frame based on the first motion vector, the optical flow, and the second image frame, wherein the second image frame is the previous image frame of the first image frame.
[0222] For example, the generation unit 1402 can be used to perform Figure 3 S303 in the method shown.
[0223] In one possible implementation, the acquisition unit 1401 is specifically used to: acquire the latent state of the second image frame; and determine the optical flow based on the latent state of the second image frame and the first motion vector.
[0224] In one possible implementation, the acquisition unit 1401 is specifically used to: determine the latent state of the second image frame based on the second motion vector of the second image frame and the latent state of the third image frame, wherein the third image frame is the previous image frame of the second image frame.
[0225] In one possible implementation, the acquisition unit 1401 is further configured to: determine the third depth information of the first image frame based on the first depth information of the second image frame and the second depth information of the third image frame;
[0226] In one possible implementation, the generation unit 1402 is specifically used to generate the first image frame based on the third depth information, the first motion vector, the optical flow, and the second image frame.
[0227] In one possible implementation, the generation unit 1402 is specifically used to: obtain the weight of the first image frame; determine the optical flow processing result based on the first motion vector, the optical flow, and the weight; and generate the first image frame based on the third depth information, the optical flow processing result, and the second image frame.
[0228] In one possible implementation, the acquisition unit 1401 is specifically used to: determine the motion vector at a second time-domain position before the first time-domain position of the first image frame in the time domain as the first motion vector.
[0229] This application also provides a chip, which can be the chip of the image generation device described above. Figure 15 A schematic diagram of a chip 1500 is shown. The chip 1500 includes one or more processors 1501 and interface circuitry 1502. Optionally, the chip 1500 may also include a bus 1503.
[0230] Processor 1501 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above image generation method can be completed through integrated logic circuits in the hardware of processor 1501 or through software instructions.
[0231] Optionally, the processor 1501 described above may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods and steps disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc.
[0232] The interface circuit 1502 can be used to send or receive data, instructions or information. The processor 1501 can use the data, instructions or other information received by the interface circuit 1502 to process the data, instructions or other information, and can send the processed information out through the interface circuit 1502.
[0233] Optionally, the chip may also include memory, which may include read-only memory and random access memory, providing operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).
[0234] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which may be stored in the operating system).
[0235] Optionally, the chip can be used in the image generation apparatus or image generation apparatus involved in the embodiments of this application. Optionally, the interface circuit 1502 can be used to output the execution result of the processor 1501. For the image generation method provided by one or more embodiments of the present application, please refer to the foregoing embodiments, which will not be repeated here.
[0236] It should be noted that the functions of processor 1501 and interface circuit 1502 can be implemented through hardware design, software design, or a combination of hardware and software; no restrictions are imposed here.
[0237] Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device can be an image generating apparatus, a chip within the image generating apparatus, or a functional module. For example... Figure 16 As shown, the electronic device 1600 includes a processor 1601, a transceiver 1602, and a communication line 1603.
[0238] The processor 1601 is used to execute any step of the image generation method provided in the embodiments of this application, and in the process of executing any step of the image generation method provided in the embodiments of this application, it can selectively call the transceiver 1602 and the communication line 1603 to complete the corresponding operation.
[0239] Furthermore, the electronic device 1600 may also include a memory 1604. The processor 1601, the memory 1604, and the transceiver 1602 can be connected via a communication line 1603.
[0240] The processor 1601 can be a processor, a general-purpose processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 1601 can also be other devices with processing capabilities, such as circuits, devices, or software modules, without limitation.
[0241] Transceiver 1602 is used to communicate with other devices or other communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. Transceiver 1602 can be a module, circuit, transceiver, or any device capable of enabling communication.
[0242] The transceiver 1602 is mainly used for sending and receiving commands and information, and may include a transmitter and a receiver to send and receive commands and information, respectively; operations other than sending and receiving commands and information are implemented by the processor.
[0243] Communication line 1603 is used to transmit information between the various components included in electronic device 1600.
[0244] In one design, the processor can be viewed as a logic circuit, and the transceiver as an interface circuit.
[0245] Memory 1604 is used to store instructions. These instructions can be computer programs.
[0246] The memory 1604 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus DRAM (DRRAM). Memory 1604 can also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices. It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0247] It should be noted that the memory 1604 can exist independently of the processor 1601, or it can be integrated with the processor 1601. The memory 1604 can be used to store instructions, program code, or some data, etc. The memory 1604 can be located inside or outside the electronic device 1600, without limitation. The processor 1601 is used to execute the instructions stored in the memory 1604 to implement the methods provided in the embodiments of this application.
[0248] In one example, processor 1601 may include one or more processor cores, for example Figure 16 The processor cores are 0 and 1.
[0249] As an optional implementation, the electronic device 1600 includes multiple processors, for example, besides Figure 16 In addition to processor 1601, it may also include processor 1607.
[0250] As an optional implementation, the electronic device 1600 also includes an output device 1605 and an input device 1606. For example, the input device 1606 is a device such as a keyboard, mouse, microphone, or joystick, and the output device 1605 is a device such as a display screen or speaker.
[0251] It should be noted that the electronic device 1600 can be a chip system or... Figure 16 Devices with similar structures. The chip system can be composed of chips or include chips and other discrete components. Actions, terminology, etc., involved in the various embodiments of this application can be referenced interchangeably without limitation. The message names or parameter names in the messages used for interaction between devices in the embodiments of this application are merely examples; other names can be used in specific implementations without limitation. Furthermore, Figure 16 The structural composition shown does not constitute a limitation on the electronic device 1600, except... Figure 16 In addition to the components shown, the electronic device 1600 may include more than Figure 16 This may indicate more or fewer components, or a combination of certain components, or a different arrangement of components.
[0252] The processor and transceiver described in this application embodiment can be implemented on integrated circuits (ICs), analog ICs, radio frequency integrated circuits, mixed-signal ICs, application-specific integrated circuits (ASICs), printed circuit boards (PCBs), electronic devices, etc. The processor and transceiver can also be manufactured using various IC process technologies, such as complementary metal-oxide-semiconductor (CMOS), n-type metal-oxide-semiconductor (NMOS), p-type metal-oxide-semiconductor (PMOS), bipolar junction transistors (BJTs), bipolar CMOS (BiCMOS), silicon germanium (SiGe), gallium arsenide (GaAs), etc.
[0253] This application also provides an image generation apparatus, which includes at least one processor. When the at least one processor executes program code or instructions, it implements the image generation method described above by executing the relevant method steps.
[0254] Optionally, the device may further include at least one memory for storing the program code or instructions.
[0255] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an image generating apparatus, the image generating apparatus performs the aforementioned method steps to implement the image generating method described above.
[0256] In one possible implementation, the computer-readable storage medium is a non-transitory computer-readable medium.
[0257] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the image generation method described above.
[0258] This application also provides an image generation apparatus, which may specifically be a chip, integrated circuit, component, or module. Specifically, the apparatus may include a connected processor and a memory for storing instructions, or the apparatus may include at least one processor for fetching instructions from external memory. When the apparatus is running, the processor can execute the instructions to cause the chip to perform the image generation methods described in the above method embodiments.
[0259] It should be understood that in various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.
[0260] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.
[0261] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0262] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0263] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0264] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0265] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this application embodiment, essentially, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0266] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be included within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.
Claims
1. An image generation method, characterized in that, include: Obtain the first motion vector of the first image frame to be generated; Obtain the optical flow of the first image frame; The first image frame is generated based on the first motion vector, the optical flow, and the second image frame, wherein the second image frame is the image frame preceding the first image frame.
2. The method according to claim 1, characterized in that, The step of obtaining the optical flow of the first image frame includes: Obtain the latent encoded state of the second image frame; The optical flow is determined based on the latent state of the second image frame and the first motion vector.
3. The method according to claim 2, characterized in that, The step of obtaining the latent state of the second image frame includes: The latent state of the second image frame is determined based on the second motion vector of the second image frame and the latent state of the third image frame, wherein the third image frame is the image frame preceding the second image frame.
4. The method according to claim 3, characterized in that, The method further includes: The third depth information of the first image frame is determined based on the first depth information of the second image frame and the second depth information of the third image frame. The step of generating the first image frame based on the first motion vector, the optical flow, and the second image frame includes: The first image frame is generated based on the third depth information, the first motion vector, the optical flow, and the second image frame.
5. The method according to claim 4, characterized in that, The step of generating the first image frame based on the third depth information, the first motion vector, the optical flow, and the second image frame includes: Obtain the weight of the first image frame; The optical flow processing result is determined based on the first motion vector, the optical flow, and the weight. The first image frame is generated based on the third depth information, the optical flow processing result, and the second image frame.
6. The method according to any one of claims 1 to 5, characterized in that, The process of obtaining the first motion vector includes: The motion vector at a second time-domain position preceding the first time-domain position of the first image frame is determined as the first motion vector.
7. An image generation apparatus, characterized in that, include: Acquisition unit and generation unit; The acquisition unit is used to acquire the first motion vector of the first image frame to be generated; The acquisition unit is further configured to acquire the optical flow of the first image frame; The generation unit is configured to generate the first image frame based on the first motion vector, the optical flow, and the second image frame, wherein the second image frame is the image frame preceding the first image frame.
8. The apparatus according to claim 7, characterized in that, The acquisition unit is specifically used for: Obtain the latent state of the second image frame; The optical flow is determined based on the latent state of the second image frame and the first motion vector.
9. The apparatus according to claim 8, characterized in that, The acquisition unit is specifically used for: The latent state of the second image frame is determined based on the second motion vector of the second image frame and the latent state of the third image frame, wherein the third image frame is the image frame preceding the second image frame.
10. The apparatus according to claim 9, characterized in that, The acquisition unit is also used for: The third depth information of the first image frame is determined based on the first depth information of the second image frame and the second depth information of the third image frame. The generation unit is specifically used for: The first image frame is generated based on the third depth information, the first motion vector, the optical flow, and the second image frame.
11. The apparatus according to claim 10, characterized in that, The generation unit is specifically used for: Obtain the weight of the first image frame; The optical flow processing result is determined based on the first motion vector, the optical flow, and the weight. The first image frame is generated based on the third depth information, the optical flow processing result, and the second image frame.
12. The apparatus according to any one of claims 7 to 11, characterized in that, The acquisition unit is specifically used for: The motion vector at a second time-domain position preceding the first time-domain position of the first image frame is determined as the first motion vector.
13. An image generation apparatus, comprising at least one processor and a memory, characterized in that, The at least one processor executes a program or instructions stored in a memory to cause the image generating apparatus to implement the method of any one of claims 1 to 6.
14. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is run on a computer or processor, it causes the computer or processor to perform the method of any one of claims 1 to 6.
15. A computer program product, the computer program product comprising instructions, characterized in that, When the instructions are executed on a computer or processor, the computer or processor causes the computer or processor to perform the method of any one of claims 1 to 6.
