Picture display method and device and related equipment

Abstract

The invention provides a picture display method and device and related equipment, and relates to the technical field of cloud computers.The method comprises the steps that under the condition that the Nth frame of image in a video stream sent by a server side is received, the (N-1) th frame of image currently displayed by the client side is obtained, the video stream comprises M frames of images, the Nth frame of image is the next frame of image of the (N-1) th frame of image in the M frames of images, M is an integer, N is an integer smaller than or equal to M, and N is larger than or equal to 2; generating a middle frame image between the (N-1) th frame image and the Nth frame image by fusing the features of the (N-1) th frame image and the features of the Nth frame image; and after replacing the (N-1) th frame of image with the middle frame of image for picture display, replacing the middle frame of image with the Nth frame of image for picture display. According to the invention, the situation that the client is blocked and unsmooth when displaying the video stream picture is reduced, and the picture display effect is improved, so that the overall use experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a screen display method, device and related equipment. Background Art

[0002] Cloud computing is a cloud computing-based service that allows users to access the entire operating system and applications on a remote server through the Internet. Users can use server-side applications and data on any client without relying on local hardware, providing flexibility and scalability. Cloud computing's video transmission protocol usually performs complex image compression algorithms on video streams to meet the needs of efficient network transmission between the client and the server.

[0003] However, during the image compression process, some image frames in the video stream will be compressed or discarded, causing the client to easily experience freezes and stuttering when displaying the video stream, resulting in poor image display effects. Summary of the invention

[0004] The embodiments of the present invention provide a screen display method, device and related equipment to solve the problem of poor screen display effect on the existing client.

[0005] To solve the above technical problems, the present invention is achieved as follows:

[0006] In a first aspect, an embodiment of the present invention provides a screen display method, which is applied to a client, and the method includes:

[0007] When receiving the Nth frame image in the video stream sent by the server, obtaining the N-1th frame image currently displayed by the client, the video stream includes M frames of images, the Nth frame image is the next frame image of the N-1th frame image in the M frames of images, M is an integer, N is an integer less than or equal to M, and N is greater than or equal to 2;

[0008] Generate an intermediate frame image between the N-1th frame image and the Nth frame image by fusing the features of the N-1th frame image and the features of the Nth frame image;

[0009] After the N-1th frame image is replaced by the intermediate frame image for screen display, the intermediate frame image is replaced by the Nth frame image for screen display.

[0010] In a second aspect, an embodiment of the present invention further provides a picture display device, the device comprising:

[0011] an acquisition module, configured to acquire, upon receiving an Nth frame image in a video stream sent by a server, an N-1th frame image currently displayed by a client, wherein the video stream includes M frames of images, the Nth frame image is a next frame image of the N-1th frame image in the M frames of images, M is an integer, N is an integer less than or equal to M, and N is greater than or equal to 2;

[0012] A generating module, configured to generate an intermediate frame image between the N-1th frame image and the Nth frame image by fusing the features of the N-1th frame image and the features of the Nth frame image;

[0013] The display module is used for replacing the N-1th frame image with the intermediate frame image for screen display, and then replacing the intermediate frame image with the Nth frame image for screen display.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device, including a transceiver and a processor.

[0015] The transceiver is used to obtain the N-1th frame image currently displayed by the client when receiving the Nth frame image in the video stream sent by the server, the video stream includes M frames of images, the Nth frame image is the next frame image of the N-1th frame image in the M frames of images, M is an integer, N is an integer less than or equal to M, and N is greater than or equal to 2;

[0016] The processor is used to generate an intermediate frame image between the N-1th frame image and the Nth frame image by fusing the features of the N-1th frame image and the features of the Nth frame image;

[0017] The processor is further configured to replace the N-1th frame image with the intermediate frame image for screen display, and then replace the intermediate frame image with the Nth frame image for screen display.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the screen display method described in the first aspect are implemented.

[0019] In a fifth aspect, an embodiment of the present invention further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the screen display method as described in the first aspect.

[0020] In an embodiment of the present invention, a screen display method applied to a client is provided, and the video stream transmitted from the server to the client may be M frame images obtained after compression processing to improve transmission efficiency; when the client receives the Nth frame image among the M frame images, an intermediate frame image can be generated according to the Nth frame image and the currently displayed N-1th frame image, and the intermediate frame image combines the features of the Nth frame image and the N-1th frame image. Before displaying the received Nth frame image, the intermediate frame image is first displayed on the screen, so that there is a smooth transition from the N-1th frame image to the Nth frame image during the display process, thereby reducing the freeze and unsmoothness of the client when displaying the video stream screen, improving the screen display effect, and thus improving the overall user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0022] Figure 1 This is one of the flow charts of a screen display method provided by an embodiment of the present invention;

[0023] Figure 2 This is a second flow chart of a screen display method provided by an embodiment of the present invention;

[0024] Figure 3 is a structural schematic diagram of a RIFE model provided by an embodiment of the present invention;

[0025] Figure 4 is a structural schematic diagram of a picture display device provided by an embodiment of the present invention;

[0026] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] See also Figure 1 , Figure 1 is one of the flowcharts of a screen display method provided by an embodiment of the present invention, which is executed by the client, such as Figure 1 As shown, the method comprises the following steps:

[0029] Step 101: upon receiving an Nth frame image in a video stream sent by a server, obtaining an N-1th frame image currently displayed by the client, wherein the video stream includes M frames of images, and the Nth frame image is a next frame image of the N-1th frame image in the M frames of images, where M is an integer, N is an integer less than or equal to M, and N is greater than or equal to 2;

[0030] Client, also known as cloud computer client. Client can be installed on various devices, such as personal computers, tablets or mobile phones. After logging in to the client, users can connect to the remote server through the Internet to access and operate the resources provided by the remote server. The client can process user input, receive and display the video stream transmitted by the server on the display interface.

[0031] For example, to meet the needs of efficient network transmission between the client and the server, the server can compress the video stream, and the video stream processed by the image compression algorithm can include M frames of images, and the server can transmit the M frames of images to the client in time sequence to achieve screen display. It should be understood that the M frames of images are images in the original video stream with some transition animation frames discarded.

[0032] When the client receives the Nth frame image in the video stream, it obtains the N-1th frame image currently displayed by the client, wherein the N-1th frame image is any frame image in the M frames image, and the Nth frame image is the next frame image after the N-1th frame image in the M frames image, that is, the Nth frame image and the N-1th frame image are adjacent time sequence images. By obtaining the image currently displayed by the client and analyzing and calculating the image to be displayed in the next frame after the currently displayed image, an intermediate frame (i.e., transition frame) image between the two frames is generated, and then the screen first displays the intermediate frame image, and then displays the next frame image sent by the server, the frame rate of the picture can be doubled, thereby improving the smoothness of the picture.

[0033] In some optional examples, when the client receives the first frame image in the video stream, that is, when N is equal to 1, and the client currently has no image to display, the client can directly display the first frame image on the screen. When the next frame image after the first frame image is received, the first frame image currently displayed by the client is obtained, and the first frame image and the next frame image after the first frame image are combined to perform analysis and calculation to generate an intermediate frame image between the two frames of images. After that, the screen first displays the intermediate frame image, and then displays the next frame image after the first frame image sent by the server, thereby improving the smoothness of the picture.

[0034] The picture scene of the image data corresponding to the video stream includes at least one of an application screen, a video content screen, and a user operation screen. Different types of picture scenes provide multi-dimensional application potential for video streams. In a variety of picture scenes, the corresponding intermediate frame image can be generated by combining the previous and next two frames of images according to different scenes to improve the smoothness of the picture.

[0035] Step 102: generating an intermediate frame image between the N-1th frame image and the Nth frame image by fusing the features of the N-1th frame image and the features of the Nth frame image;

[0036] The intermediate frame image generated based on the N-1 frame image and the N frame image combines the features of the N-1 frame image and the N frame image. In this way, the intermediate frame not only retains the continuity of the previous frame, but also introduces new elements of the next frame, reducing abrupt visual switching and improving the overall smoothness and visual quality of the picture.

[0037] Optionally, the step 102 of generating an intermediate frame image between the N-1th frame image and the Nth frame image by fusing the features of the N-1th frame image and the features of the Nth frame image includes:

[0038] The N-1th frame image and the Nth frame image are input into a pre-built real-time intermediate flow estimation algorithm RIFE model to generate an intermediate frame image between the N-1th frame image and the Nth frame image.

[0039] In this embodiment, the N-1th frame image and the Nth frame image are analyzed and calculated through a pre-built real-time intermediate flow estimation algorithm (RIFE) model to generate an intermediate frame image between the two frames. This can effectively create a high-quality intermediate frame image and achieve a smooth transition between the previous frame (i.e., the N-1th frame) and the next frame (i.e., the N-th frame), making the picture display more coherent and natural, thereby improving the overall user experience of the cloud computer.

[0040] Specifically, the N-1th frame and the Nth frame image are passed as input to the RIFE model; the RIFE model can extract deep features from the two input frames, for example, using a convolutional neural network (CNN) to capture key details and structural information in the frame; and calculate the optical flow between the N-1th frame and the Nth frame, that is, the motion trajectory of each pixel; combining the features of the N-1th frame and the Nth frame, an intermediate frame image is generated through an interpolation algorithm, and the intermediate frame image effectively integrates the features from the previous and next frames, so that the display process from the N-1th frame image to the Nth frame image has a smooth transition, which reduces the freeze and unsmoothness of the client when displaying the video stream, and improves the picture display effect.

[0041] Step 103: After replacing the N-1th frame image with the intermediate frame image for screen display, replacing the intermediate frame image with the Nth frame image for screen display.

[0042] After generating the intermediate frame image between the N-1th frame image and the Nth frame image, the N-1th frame image is first replaced with the intermediate frame image for screen display; after displaying an intermediate frame image, the client waits for an inter-frame time interval (which can be the inverse of the video frame rate) and then replaces the intermediate frame image with the Nth frame image for screen display. In this way, the intermediate frame image fills the transition frame lost in the compression process between the N-1th frame image and the Nth frame image in the video stream, and alternately displays the original video image (i.e., the N-1th frame image and the Nth frame image) and the generated video image (i.e., the intermediate frame image between the N-1th frame image and the Nth frame image), thereby achieving smooth display of the video, allowing the client to present a more smooth and natural cloud computer screen display effect.

[0043] In an embodiment of the present invention, a screen display method applied to a client is provided, and the video stream transmitted from the server to the client may be M frame images obtained after compression processing to improve transmission efficiency; when the client receives the Nth frame image among the M frame images, an intermediate frame image can be generated according to the Nth frame image and the currently displayed N-1th frame image, and the intermediate frame image combines the features of the Nth frame image and the N-1th frame image. Before displaying the received Nth frame image, the intermediate frame image is first displayed on the screen, so that there is a smooth transition from the N-1th frame image to the Nth frame image during the display process, thereby reducing the freeze and unsmoothness of the client when displaying the video stream screen, improving the screen display effect, and thus improving the overall user experience.

[0044] The features of the N-1th frame image and the Nth frame image can be fused through the RIFE model to generate an intermediate frame image. For example, the pre-trained RIFE model can be embedded in the cloud computer client, and the RIFE model generates a transitional intermediate frame image based on the previous frame image (i.e., the N-1th frame image) and the next frame image (i.e., the Nth frame image). Figure 2 As shown, after the client receives the video stream sent by the service, it decodes the video stream to obtain M frames of images; when the Nth frame image is read, the N-1th frame image currently displayed by the client is obtained (that is, the previous frame image of the Nth frame image); then the Nth frame image and the N-1th frame image are input into the configured RIFE model together to generate the intermediate frame image between the N-1th frame image and the Nth frame image; after the N-1th frame image is replaced with the intermediate frame image for screen display, the intermediate frame image is replaced with the Nth frame image for screen display. This reduces the situation of freeze and unsmoothness when the client displays the video stream screen, and improves the screen display effect.

[0045] The training of the RIFE model can be achieved by building a dedicated training set, collecting computer images with complete transition animations on traditional computers, and building a training set to make the algorithm model more suitable for the use scenarios of cloud computers. For details, please refer to the following description:

[0046] Optionally, the construction process of the RIFE model includes the following steps:

[0047] Obtaining first model parameters corresponding to the first model after training based on the training sample set, wherein the first model is a model obtained by updating the preset initial model in the i-th iteration, and the preset initial model includes an image forming module, an image deformation module and an image fusion module, and the preset initial model is used to generate an intermediate frame image between adjacent image frames, and i is a positive integer;

[0048] The model parameters of the first model are updated based on the first model parameters to obtain the RIFE model, wherein the RIFE model is a model obtained by updating the preset initial model at the i+1th iteration.

[0049] In this embodiment, if Figure 3As shown, the preset initial model includes an image formation module (IFnet), an image deformation module (Backward Warping) and an image fusion module. The image formation module is used for feature extraction and image generation, and generates new frames by integrating the features of the previous and next frames; the image deformation module is used to achieve inter-frame mapping, and accurately establishes the relationship between the intermediate frame and the adjacent frames through reverse mapping, which helps to improve the overall generation quality; the image fusion module is used to integrate the extracted features, which helps to make full use of the information of the two frames of images. During the image synthesis process, the image fusion module can effectively identify and remove possible artifacts, making the final output more natural and clear. After the two initial frames I0 and I1 are processed by the image formation module, the image deformation module and the image fusion module respectively, all the data are fused to obtain the predicted intermediate frame I t The features of adjacent images are integrated and processed to create higher quality and more natural intermediate frames, effectively improving the visual effects and user experience of the entire video stream.

[0050] Among them, the construction process of the RIFE model is actually an iterative process of the corresponding parameters of the image formation module, the image deformation module and the image fusion module. In other words, the first model parameters may include the parameters of the image formation module, the parameters of the image deformation module and the parameters of the image fusion module.

[0051] Specifically, first, a training sample set containing multiple pairs of adjacent image frames is collected and prepared to provide a learning basis for the model. The training sample set can be a series of continuous original video frames or image sequences collected from traditional computers, ensuring that the collected video frames or image sequences cover various scenarios of cloud computer use.

[0052] Optionally, before obtaining the first model parameters corresponding to the first model after training the first model based on the training sample set, the method further includes:

[0053] Acquire first image data having continuous frames, wherein the picture scenes of the first image data include application-type pictures, video content-type pictures, and user operation-type pictures;

[0054] Performing a transformation process on the first image data to obtain second image data, wherein the amount of data of the second image data is greater than the amount of data of the first image data, the transformation process comprising at least one of a frame rate transformation process and a resolution transformation process;

[0055] The second image data is preprocessed to obtain the training sample set, wherein the preprocessing includes at least one of scaling, cropping, denoising, contrast enhancement and normalization.

[0056] Exemplarily, the first image data is a video clip containing continuous, rapid and dynamic changes, and the screen scenes of the first image data include application screens, video content screens and user operation screens to simulate the real-time and diversity characteristics of cloud computer screens. Among them, application screens can include screens of office programs, entertainment programs and design programs. Office programs can include word processing programs (such as Microsoft Word, Google Docs and LibreOffice Writer), spreadsheet programs (such as Microsoft Excel, Google Sheets and LibreOffice Calc), presentation software programs (such as Microsoft PowerPoint, Google Slides and LibreOffice Impress), email programs (such as Microsoft Outlook, Mozilla Thunderbird and Gmail), project management programs (such as Microsoft Project, Trello and Asana), cloud storage and file sharing programs (such as Google Drive, Dropbox and OneDrive). Entertainment programs may include video streaming programs (such as Netflix, YouTube, and Hulu), music playing programs (such as Spotify, Apple Music, and Pandora), game programs (such as Steam, Epic Games Store, and GOG), social media programs (such as Facebook, Instagram, and TikTok), and e-book and magazine programs (such as Kindle, Apple Books, and Libby). Design programs may include graphic design programs (such as Adobe Photoshop, Adobe Illustrator, and CorelDRAW), video editing programs (such as Adobe Premiere Pro, FinalCut Pro, and DaVinci Resolve), typesetting and publishing programs (such as Adobe InDesign, QuarkXPress, and Scribus), web design programs (such as Webflow, WordPress, and Adobe Dreamweaver), and interior design programs (such as AutoCAD, SketchUp, and Revit). Video content screens may include screens of movie content, animation content, and live content. User operation screens may include screens of mouse clicks, dragged screens, and scrolled screens.

[0057] Then, the first image data is transformed to obtain second image data, the data volume of the second image data is greater than the data volume of the first image data, and the transformation process includes at least one of frame rate transformation and resolution transformation to expand the number of training samples. Exemplarily, the cloud computer service can support multiple frame rates and resolutions. Therefore, the training set may include video clips with different frame rates and resolutions to ensure that the model can accurately generate interpolated frames under various configurations. For example, the training set may contain video clips ranging from 30fps to 120fps, and materials with different resolutions ranging from 1080p to 4K.

[0058] The second image data is then preprocessed to obtain a training sample set, and the preprocessing includes at least one of scaling, cropping, denoising, contrast enhancement and normalization. Scaling adjusts the size of the image to meet different display requirements or algorithm requirements; cropping removes unnecessary parts of the image to highlight the area of ​​interest; denoising reduces random noise in the image to make the image clearer and more realistic; contrast enhancement increases the difference between light and dark in the image to make the image more vivid and eye-catching; and normalization normalizes the pixel value of the image to a specific range (such as 0 to 1 or -1 to 1) for subsequent processing.

[0059] Traverse each frame I from front to back for all frames in the training sample set n , and the I (n+2) Together as a reference frame as input, the I (n+1) Frame is used as the target frame. All reference frames and target frames are divided into training set and validation set. During the training process, the reference frame is input into the RIFE algorithm model from the training set to train the first model, and the target frame is used as the evaluation benchmark to measure the model's ability to generate intermediate frames, and the relevant parameters of the model are dynamically updated according to the performance of these target frames. The validation set is used to comprehensively evaluate the interpolation performance of the model. When the model can meet the preset generation quality standard, the training process is stopped to obtain the first model parameters corresponding to the first model, where the first model is the model obtained by updating the preset initial model at the i-th iteration; and the model parameters of the first model are updated based on the first model parameters to obtain the RIFE model, where the RIFE model is the model obtained by updating the preset initial model at the i+1-th iteration. This training and update mechanism can effectively improve the accuracy and robustness of the RIFE model, making it perform well in intermediate frame generation.

[0060] In this way, an intermediate frame is generated using the trained RIFE model. Subsequently, the client can first display the intermediate frame for a period of time according to the preset frame rate, and then smoothly transition to the first frame image, thereby achieving a continuous and smooth picture. It is worth noting that each reception of the first frame image will trigger the generation of the corresponding intermediate frame image, and the Nth frame image will then become one of the reference frames for the N+1th frame image to generate the intermediate frame. In this way, the present invention can not only effectively fill the time gap between frames, but also ensure the stability and continuity of the picture in real-time transmission, bringing a more natural and smooth visual experience to cloud computer users.

[0061] Optionally, before inputting the N-1th frame image and the Nth frame image into a pre-built real-time intermediate flow estimation algorithm RIFE model to generate an intermediate frame image between the N-1th frame image and the Nth frame image, the method further includes:

[0062] Obtaining a pre-built RIFE model, and reading the device information of the client and the picture scene information of the video stream;

[0063] According to the device information and the picture scene information, a target connection layer in the RIFE model is pruned, the target connection layer being a connection layer in the connection layer of the RIFE model whose weight value is less than a preset weight threshold.

[0064] In this embodiment, the RIFE model can also be optimized according to the running device and scenario of the RIFE model. First, obtain the pre-built RIFE model, and read the device information of the client and the picture scene information of the video stream. The device information may include the supported resolution, frame rate and other information of the display device. The picture scene includes application class pictures, video content class pictures and user operation class pictures. Then, according to the device information and picture scene information, prune the target connection layer in the RIFE model. The target connection layer is the connection layer in the connection layer of the RIFE model whose weight value is less than the preset weight threshold. In this way, through the above steps, the RIFE model can be effectively optimized, and unimportant parts of the model, such as connections with smaller weights, can be removed to reduce the model size and improve the reasoning efficiency, and quantization can be used to reduce the representation accuracy of the model weights and activation values, significantly reducing the model size and speeding up the reasoning speed.

[0065] In addition, in the development process of the cloud computer client, in order to meet the demand for real-time interpolation of video frames, the trained RIFE model can be embedded into the client's image output module. Specifically, the integration of the deep learning framework and the implementation of the model inference function can be utilized.

[0066] First, select the same deep learning framework or a compatible inference engine as when building the RIFE model to ensure the compatibility and accuracy of the model during inference. Then, embed the exported model file into the binary file or resource folder of the client application so that it can be automatically loaded when the client starts. When the client starts, use the selected inference engine to load and initialize the embedded RIFE model. This includes loading the model's weights and configuration files, as well as performing some necessary initial settings such as allocating memory, setting the shapes of input and output tensors, etc., to ensure that the model can run correctly on the client and is ready to accept input data. Once the model is loaded and initialized, the client can start receiving the video stream sent from the server. For the first frame of the image received, it can be sent directly to the graphics display module for display so that the user can see the starting frame of the video, and the intermediate frames are generated starting from the second frame of the image.

[0067] When the client receives the second and subsequent video frames, it will first use the loaded and initialized RIFE model for inference to generate intermediate frames. These intermediate frames are calculated by the model based on the information of the previous and next frames. The intermediate frames can fill the time interval between the original video frames, thereby achieving smooth display of the video. The generated intermediate frames are then sent to the graphics display module for display. After displaying an intermediate frame, the client will wait for an inter-frame time interval and then display the next original video frame received. In this way, by alternating the display of intermediate frames and original frames, the client can present a smoother and more natural cloud computer screen display effect.

[0068] Optionally, the method further comprises:

[0069] Reading resource information of the client, the resource information including operation data of at least one of a central processing unit, an image processing unit, and a memory;

[0070] When the resource information indicates that the client load is greater than a preset load threshold, when the Lth frame image sent by the server is received, the L-1th frame image is replaced with the Lth frame image for screen display, and the Lth frame image is the next frame image after the L-1th frame image in the M frame images, L is a positive integer less than or equal to M, and L is greater than or equal to 2.

[0071] In this embodiment, in order to more flexibly use an image display method provided by an embodiment of the present invention and adapt to different usage scenarios and hardware conditions, a cloud computer client resource monitoring module can also be set. This module will monitor the resource information of the client in real time, such as the operating data of the central processing unit, image processor and memory. When the resource information indicates that the client load is greater than the preset load threshold, that is, when the computing resources of the client are too few, the resource monitoring module will trigger the corresponding optimization strategy. When the L-th frame image sent by the server is received, the L-1th frame image is directly replaced with the L-th frame image for screen display, that is, the frame interpolation operation between the L-1th frame image and the L-th frame image is suspended.

[0072] It should be understood that for M-frame images, the number of intermediate frame images generated can be reduced to ensure that the client can still maintain smooth operation under resource constraints and avoid problems such as freezes or crashes due to insufficient resources. By embedding the RIFE model into the video output module of the cloud computer client and combining the integration of the deep learning framework and the inference engine, real-time interpolation of video frames is successfully achieved. At the same time, through the optimization strategy of the resource monitoring module, it can more flexibly respond to different usage scenarios and hardware conditions to ensure the stability and smoothness of the client. It reduces the freeze and unsmoothness of the client when displaying the video stream, and improves the picture display effect.

[0073] See also Figure 4 , Figure 4 is a structural schematic diagram of a picture display device provided by an embodiment of the present invention, such as Figure 4 As shown, the screen display device 400 includes:

[0074] The acquisition module 401 is used to acquire the N-1th frame image currently displayed by the client when the Nth frame image in the video stream sent by the server is received, the video stream includes M frames of images, the Nth frame image is the next frame image of the N-1th frame image in the M frames of images, M is an integer, N is an integer less than or equal to M, and N is greater than or equal to 2;

[0075] A generating module 402, configured to generate an intermediate frame image between the N-1th frame image and the Nth frame image by fusing the features of the N-1th frame image and the features of the Nth frame image;

[0076] The display module 403 is used to replace the N-1th frame image with the intermediate frame image for screen display, and then replace the intermediate frame image with the Nth frame image for screen display.

[0077] Optionally, the generating module 402 is specifically configured to:

[0078] The N-1th frame image and the Nth frame image are input into a pre-built real-time intermediate flow estimation algorithm RIFE model to generate an intermediate frame image between the N-1th frame image and the Nth frame image.

[0079] Optionally, the construction process of the RIFE model includes the following steps:

[0080] Obtaining first model parameters corresponding to the first model after training based on the training sample set, wherein the first model is a model obtained by updating the preset initial model in the i-th iteration, and the preset initial model includes an image forming module, an image deformation module and an image fusion module, and the preset initial model is used to generate an intermediate frame image between adjacent image frames, and i is a positive integer;

[0081] The model parameters of the first model are updated based on the first model parameters to obtain the RIFE model, wherein the RIFE model is a model obtained by updating the preset initial model at the i+1th iteration.

[0082] Optionally, before obtaining the first model parameters corresponding to the first model after training the first model based on the training sample set, the method further includes:

[0083] Acquire first image data having continuous frames, wherein the picture scenes of the first image data include application-type pictures, video content-type pictures, and user operation-type pictures;

[0084] Performing a transformation process on the first image data to obtain second image data, wherein the amount of data of the second image data is greater than the amount of data of the first image data, the transformation process comprising at least one of a frame rate transformation process and a resolution transformation process;

[0085] The second image data is preprocessed to obtain the training sample set, wherein the preprocessing includes at least one of scaling, cropping, denoising, contrast enhancement and normalization.

[0086] Optionally, the generating module 402 is further configured to:

[0087] Obtaining a pre-built RIFE model, and reading the device information of the client and the picture scene information of the video stream;

[0088] According to the device information and the picture scene information, a target connection layer in the RIFE model is pruned, the target connection layer being a connection layer in the connection layer of the RIFE model whose weight value is less than a preset weight threshold.

[0089] Optionally, the device further comprises:

[0090] A reading module, used for reading resource information of the client, wherein the resource information includes operation data of at least one of a central processing unit, an image processing unit and a memory;

[0091] A receiving module is used to replace the L-1 frame image with the L-1 frame image for screen display when the L-1 frame image sent by the server is received, when the resource information indicates that the client load is greater than a preset load threshold. The L-1 frame image is the next frame image of the L-1 frame image in the M frame images, L is a positive integer less than or equal to M, and L is greater than or equal to 2.

[0092] Optionally, the picture scene of the image data corresponding to the video stream includes at least one of an application-type picture, a video content-type picture, and a user operation-type picture.

[0093] It should be noted that the screen display device 400 is capable of implementing each process of each embodiment of the above-mentioned screen display method, the technical features correspond one to one, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0094] An embodiment of the present invention also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, each process of the above-mentioned screen display method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0095] For details, see Figure 5 An embodiment of the present invention further provides an electronic device, including a bus 501, a transceiver 502, an antenna 503, a bus interface 504, a processor 505 and a memory 506.

[0096] The transceiver 502 is used to obtain the N-1th frame image currently displayed by the client when receiving the Nth frame image in the video stream sent by the server, the video stream includes M frames of images, the Nth frame image is the next frame image of the N-1th frame image in the M frames of images, M is an integer, N is an integer less than or equal to M, and N is greater than or equal to 2;

[0097] The processor 505 is configured to generate an intermediate frame image between the N-1th frame image and the Nth frame image by fusing the features of the N-1th frame image and the features of the Nth frame image;

[0098] The processor 505 is further configured to replace the N-1th frame image with the intermediate frame image for screen display, and then replace the intermediate frame image with the Nth frame image for screen display.

[0099] Optionally, the generating an intermediate frame image between the N-1th frame image and the Nth frame image by fusing the features of the N-1th frame image and the features of the Nth frame image comprises:

[0100] The N-1th frame image and the Nth frame image are input into a pre-built real-time intermediate flow estimation algorithm RIFE model to generate an intermediate frame image between the N-1th frame image and the Nth frame image.

[0101] Optionally, the construction process of the RIFE model includes the following steps:

[0102] Obtaining first model parameters corresponding to the first model after training based on the training sample set, wherein the first model is a model obtained by updating the preset initial model in the i-th iteration, and the preset initial model includes an image forming module, an image deformation module and an image fusion module, and the preset initial model is used to generate an intermediate frame image between adjacent image frames, and i is a positive integer;

[0103] The model parameters of the first model are updated based on the first model parameters to obtain the RIFE model, wherein the RIFE model is a model obtained by updating the preset initial model at the i+1th iteration.

[0104] Optionally, before obtaining the first model parameters corresponding to the first model after training the first model based on the training sample set, the method further includes:

[0105] Acquire first image data having continuous frames, wherein the picture scenes of the first image data include application-type pictures, video content-type pictures, and user operation-type pictures;

[0106] Performing a transformation process on the first image data to obtain second image data, wherein the amount of data of the second image data is greater than the amount of data of the first image data, the transformation process comprising at least one of a frame rate transformation process and a resolution transformation process;

[0107] The second image data is preprocessed to obtain the training sample set, wherein the preprocessing includes at least one of scaling, cropping, denoising, contrast enhancement and normalization.

[0108] Optionally, before inputting the N-1th frame image and the Nth frame image into a pre-built real-time intermediate flow estimation algorithm RIFE model to generate an intermediate frame image between the N-1th frame image and the Nth frame image, the method further includes:

[0109] Obtaining a pre-built RIFE model, and reading the device information of the client and the picture scene information of the video stream;

[0110] According to the device information and the picture scene information, a target connection layer in the RIFE model is pruned, the target connection layer being a connection layer in the connection layer of the RIFE model whose weight value is less than a preset weight threshold.

[0111] Optionally, the transceiver 502 is further used to read resource information of the client, wherein the resource information includes operation data of at least one of a central processing unit, an image processor and a memory;

[0112] The processor 505 is also used to replace the L-1 frame image with the L-1 frame image for screen display when the L-1 frame image sent by the server is received, when the resource information indicates that the client load is greater than a preset load threshold. The L-1 frame image is the next frame image of the L-1 frame image in the M frame images, and L is a positive integer less than or equal to M, and L is greater than or equal to 2.

[0113] Optionally, the picture scene of the image data corresponding to the video stream includes at least one of an application-type picture, a video content-type picture, and a user operation-type picture.

[0114] exist Figure 5 In the embodiment, the bus architecture (represented by bus 501) is shown, and bus 501 may include any number of interconnected buses and bridges, and bus 501 links various circuits including one or more processors represented by processor 505 and memory represented by memory 506. Bus 501 may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. Bus interface 504 provides an interface between bus 501 and transceiver 502. Transceiver 502 may be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. Data processed by processor 505 is transmitted on a wireless medium via antenna 503, and further, antenna 503 also receives data and transmits the data to processor 505.

[0115] The processor 505 is responsible for managing the bus 501 and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management and other control functions. The memory 506 can be used to store data used by the processor 505 when performing operations.

[0116] Optionally, the processor 505 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD).

[0117] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above-mentioned screen display method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0118] The embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned screen display method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0119] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present invention is not limited to performing functions in the order discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0121] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A screen display method, characterized in that: Applied to a client, the method comprises: When receiving the Nth frame image in the video stream sent by the server, obtaining the N-1th frame image currently displayed by the client, the video stream includes M frames of images, the Nth frame image is the next frame image of the N-1th frame image in the M frames of images, M is an integer, N is an integer less than or equal to M, and N is greater than or equal to 2; Generate an intermediate frame image between the N-1th frame image and the Nth frame image by fusing the features of the N-1th frame image and the features of the Nth frame image; After the N-1th frame image is replaced by the intermediate frame image for screen display, the intermediate frame image is replaced by the Nth frame image for screen display.

2. The method according to claim 1, characterized in that The step of generating an intermediate frame image between the N-1th frame image and the Nth frame image by fusing the feature of the N-1th frame image and the feature of the Nth frame image comprises: The N-1th frame image and the Nth frame image are input into a pre-built real-time intermediate flow estimation algorithm RIFE model to generate an intermediate frame image between the N-1th frame image and the Nth frame image.

3. The method according to claim 2, characterized in that The construction process of the RIFE model includes the following steps: Obtaining first model parameters corresponding to the first model after training based on the training sample set, wherein the first model is a model obtained by updating the preset initial model in the i-th iteration, and the preset initial model includes an image forming module, an image deformation module and an image fusion module, and the preset initial model is used to generate an intermediate frame image between adjacent image frames, and i is a positive integer; The model parameters of the first model are updated based on the first model parameters to obtain the RIFE model, wherein the RIFE model is a model obtained by updating the preset initial model at the i+1th iteration.

4. The method according to claim 3, characterized in that Before obtaining the first model parameters corresponding to the first model after training the first model based on the training sample set, the method further includes: Acquire first image data having continuous frames; Performing a transformation process on the first image data to obtain second image data, wherein the amount of data of the second image data is greater than the amount of data of the first image data, the transformation process comprising at least one of a frame rate transformation process and a resolution transformation process; The second image data is preprocessed to obtain the training sample set, wherein the preprocessing includes at least one of scaling, cropping, denoising, contrast enhancement and normalization.

5. The method according to claim 2, characterized in that: Before inputting the N-1th frame image and the Nth frame image into a pre-built real-time intermediate flow estimation algorithm RIFE model to generate an intermediate frame image between the N-1th frame image and the Nth frame image, the method further includes: Obtaining a pre-built RIFE model, and reading the device information of the client and the picture scene information of the video stream; According to the device information and the picture scene information, a target connection layer in the RIFE model is pruned, the target connection layer being a connection layer in the connection layer of the RIFE model whose weight value is less than a preset weight threshold.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Reading resource information of the client, the resource information including operation data of at least one of a central processing unit, an image processing unit, and a memory; When the resource information indicates that the client load is greater than a preset load threshold, when the Lth frame image sent by the server is received, the L-1th frame image is replaced with the Lth frame image for screen display, and the Lth frame image is the next frame image of the L-1th frame image in the M frame images, L is a positive integer less than or equal to M, and L is greater than or equal to 2.

7. The method according to any one of claims 1 to 5, characterized in that The picture scene of the image data corresponding to the video stream includes at least one of an application-type picture, a video content-type picture, and a user-operation-type picture.

8. A screen display device, characterized in that: The device comprises: an acquisition module, configured to acquire, upon receiving an Nth frame image in a video stream sent by a server, an N-1th frame image currently displayed by a client, wherein the video stream includes M frames of images, the Nth frame image is a next frame image of the N-1th frame image in the M frames of images, M is an integer, N is an integer less than or equal to M, and N is greater than or equal to 2; A generating module, configured to generate an intermediate frame image between the N-1th frame image and the Nth frame image by fusing the features of the N-1th frame image and the features of the Nth frame image; The display module is used for replacing the N-1th frame image with the intermediate frame image for screen display, and then replacing the intermediate frame image with the Nth frame image for screen display.

9. An electronic device, characterized in that: Including transceiver and processor, The transceiver is used to obtain the N-1th frame image currently displayed by the client when receiving the Nth frame image in the video stream sent by the server, the video stream includes M frames of images, the Nth frame image is the next frame image of the N-1th frame image in the M frames of images, M is an integer, N is an integer less than or equal to M, and N is greater than or equal to 2; The processor is used to generate an intermediate frame image between the N-1th frame image and the Nth frame image by fusing the features of the N-1th frame image and the features of the Nth frame image; The processor is further configured to replace the N-1th frame image with the intermediate frame image for screen display, and then replace the intermediate frame image with the Nth frame image for screen display.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the screen display method according to any one of claims 1 to 7 are implemented.

11. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the screen display method according to any one of claims 1 to 7.

Images