Image processing method, device and equipment

Abstract

The application discloses an image processing method, device and equipment, which can be applied to various scenes such as image processing, games and game basic technology. The method comprises the following steps: when a decoding cache image exists in a terminal, performing an interpolation operation on continuous multiple first images to generate continuous multiple second images, the interpolation operation comprises inserting N intermediate images between any two adjacent first images, the intermediate images are generated according to the first images, the decoding cache image is an image cached in a decoder of the terminal, and the value of N is determined according to the number of the decoding cache image; and decoding the second images. In the embodiment of the application, when the decoder decodes, after one first image is input into the decoder, N adjacent intermediate images are input into the decoder as decoding cache images. Since the intermediate images are generated according to the first images, the decoding time required by the intermediate images is short, so that the first image can be quickly output from the decoder, and the single-frame decoding delay can be reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to an image processing method, apparatus, and device. Background Technology

[0002] Currently, when the bitstream is sent to the chip decoder, the decoder needs to buffer several frames in the decoding buffer image before outputting the decoded frame. These buffered frames are called the decoding buffer image, or frame hoarding. If the chip exhibits frame hoarding, it generally introduces decoding latency. Cloud gaming scenarios have extremely high requirements for decoding latency, thus reducing the cloud gaming experience. Summary of the Invention

[0003] This application provides an image processing method, apparatus, and device that can effectively reduce decoding latency.

[0004] On one hand, an image processing method is provided, the method comprising, when a decoding cache image exists on a terminal, performing a frame interpolation operation on multiple consecutive frames of a first image to generate multiple consecutive frames of a second image, the frame interpolation operation comprising inserting N intermediate images between any two adjacent frames of the first image, the intermediate images being generated based on the first images, the decoding cache image being an image cached in the decoder of the terminal, the value of N being determined based on the number of decoding cache images; and decoding the second image.

[0005] On the other hand, an image processing method is provided, the method comprising, when a decoding cache image exists on the terminal, performing a frame interpolation operation on multiple consecutive first images before decoding on the terminal to generate multiple consecutive second images, the frame interpolation operation comprising inserting N intermediate images between any two adjacent first images, the intermediate images being generated based on the first images, the decoding cache image being an image cached in the decoder of the terminal, the value of N being determined based on the number of decoding cache images; and decoding the second images.

[0006] On the other hand, an image processing method is provided, the method comprising, when detecting that a terminal connected to the cloud server has a decoded cached image, performing a frame interpolation operation on multiple consecutive frames of first images during encoding on the cloud server to generate multiple consecutive frames of second images, the frame interpolation operation comprising inserting N intermediate images between any two adjacent frames of the first images, the intermediate images being generated based on the first images, the decoded cached images being images cached in the decoder of the terminal, the value of N being determined based on the number of decoded cached images; and sending the encoded multiple consecutive frames of the second images to the terminal so that the terminal can decode the second images.

[0007] On the other hand, an image processing apparatus is provided, the apparatus including a first frame interpolation module and a first decoding module. The first frame interpolation module is used to perform frame interpolation operations on multiple consecutive frames of first images when a decoded cache image exists on the terminal, to generate multiple consecutive frames of second images. The frame interpolation operation includes inserting N intermediate images between any two adjacent frames of the first images. The intermediate images are generated based on the first images. The decoded cache images are images cached in the decoder of the terminal, and the value of N is determined according to the number of decoded cache images. The first decoding module is used to decode the second images.

[0008] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the processor executing the steps of the image processing method as described in any of the above embodiments by calling the computer program stored in the memory.

[0009] This application embodiment generates a series of second images by interpolating N intermediate images generated from the first images between adjacent first images, thereby decoding the second images. During decoding, the number of frames stored in the decoder is fixed. Compared to existing solutions that input adjacent first images as decoding buffers, this approach, by inputting N adjacent intermediate images as decoding buffers, reduces the decoding time required for the intermediate images, which are generated from the first images. This allows the first images to be output from the decoder quickly, reducing single-frame decoding latency. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application.

[0012] Figure 2 This is a schematic diagram of the structure of the image processing system provided in an embodiment of this application.

[0013] Figures 3 to 13 This is a schematic flowchart of the image processing method provided in an embodiment of this application.

[0014] Figures 14 to 17 This is a schematic flowchart of an image processing method provided in another embodiment of this application.

[0015] Figures 18 to 20 This is a schematic flowchart of an image processing method provided in another embodiment of this application.

[0016] Figure 21 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application.

[0017] Figure 22 This is a schematic diagram of the structure of an image processing apparatus provided in another embodiment of this application.

[0018] Figure 23 This is a schematic diagram of the structure of an image processing apparatus provided in another embodiment of this application.

[0019] Figure 24 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] This application provides an image processing method, apparatus, computer device, and storage medium. Specifically, the image processing method of this application can be executed by a computer device, which can be a terminal or a cloud server, etc. The terminal can be a smartphone, tablet, laptop, desktop computer, smart TV, smart speaker, wearable smart device, smart vehicle terminal, etc. The terminal can also include a client, which can be a cloud gaming client, a client applet, a video client, a browser client, or an instant messaging client, etc. The cloud server can be an independent physical cloud server, a cloud server cluster or distributed system composed of multiple physical cloud servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0022] The embodiments of this application can be applied to various scenarios such as image processing, games, and game-related technologies.

[0023] For example, when this method runs on a cloud server, it can be used for cloud gaming. Cloud gaming refers to a gaming method based on cloud computing. In the cloud gaming operating mode, the main body running the game application and the main body displaying the game screen are separated. The storage and execution of the method are completed on the cloud gaming server. The display of the game screen is completed on the cloud gaming client. The cloud gaming client is mainly used for receiving and sending game data and displaying the game screen. For example, the cloud gaming client can be a display device with data transmission capabilities close to the user, such as a mobile terminal, television, computer, PDA, personal digital assistant, etc., but the terminal device for processing game data is the cloud gaming server in the cloud. When playing the game, the user operates the cloud gaming client to send operation commands to the cloud gaming server. The cloud gaming server runs the game according to the operation commands, encodes and compresses the game screen and other data, returns it to the cloud gaming client through the network, and finally, the cloud gaming client decodes and outputs the game screen.

[0024] First, some of the nouns or terms that appear in the description of the embodiments of this application are explained as follows:

[0025] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Based on the cloud computing business model, cloud technology encompasses network technology, information technology, integration technology, management platform technology, and application technology. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0026] Cloud gaming, also known as gaming on demand, is an online gaming technology based on cloud computing. It enables thin clients with relatively limited graphics processing and data processing capabilities to run high-quality games. In cloud gaming, the game does not reside on the player's terminal but runs on a cloud server. The cloud server renders the game scene as a video and audio stream, which is then transmitted to the player's terminal via the network. The player's terminal does not need powerful graphics processing and data processing capabilities; it only needs basic streaming media playback capabilities and the ability to receive player input commands and send them to the cloud server.

[0027] Video encoding: The method of converting a file in an original video format into a file in another video format through compression technology. The converted data can be called a bitstream.

[0028] Video decoding: The reverse process of video encoding.

[0029] Decoding buffered image: When the bitstream is sent to the chip decoder, the chip decoder needs to buffer several frames in the decoding buffer before outputting the decoded frame. The buffered frame is called the decoding buffer image, or the stored frame.

[0030] Single-frame decoding latency: The difference between the time a video frame enters the decoder and the time the video frame is rendered on the terminal.

[0031] P-skip frame: The decoded image is the same as the previous frame, but only a tiny bit smaller than a normal frame.

[0032] CAVLC encoding: Context-adaptive variable-length encoding.

[0033] CABAC encoding: Context-based adaptive binary arithmetic encoding.

[0034] A cloud server is a server that runs games in the cloud and has functions such as video enhancement (pre-encoding processing) and video encoding.

[0035] A terminal refers to a type of device that has rich human-computer interaction methods, internet access capabilities, typically runs various operating systems, and possesses strong processing power. Terminals include smartphones, living room TVs, tablets, in-vehicle terminals, handheld game consoles, etc.

[0036] To enable better collaboration and joint optimization between cloud servers and terminals, video encoding configurations and computational tasks such as video image processing and video content analysis can be rationally allocated based on the terminal's hardware capabilities and real-time performance during cloud gaming. This further enhances the cloud gaming visual experience within limited cloud server resources.

[0037] Video encoding collaboration: Based on the terminal's encoding and decoding capabilities, and in combination with the game type and user network type, select the optimal encoding and decoding configuration and strategy.

[0038] Video rendering collaboration: Based on the terminal's graphics processing capabilities, video rendering tasks are rationally divided to enable effective collaboration between the cloud server and the terminal, thereby improving video quality. This includes collaboration in rendering regions, rendering tasks, and video analysis and processing.

[0039] Terminal status coordination: Based on the real-time performance of the terminal, dynamically adjust the encoding coordination tasks and rendering coordination strategies of the cloud server and the terminal to ensure the best user experience in real time.

[0040] The cloud server and terminal collaborative architecture mainly includes: cloud server, terminal-cloud collaborative strategy, terminal-cloud collaborative protocol, terminal collaborative interface, and software and hardware collaborative module.

[0041] The edge-cloud collaboration strategy includes: video encoding collaboration strategy, video rendering collaboration strategy, and terminal status collaboration strategy. The cloud server formulates the optimal video encoding and rendering collaboration strategy based on the device capabilities reported by the terminal, combined with the game type and the user's network environment. Simultaneously, the cloud server dynamically adjusts the edge-cloud collaboration strategy by acquiring real-time terminal performance data through terminal status collaboration.

[0042] The edge-cloud collaboration protocol refers to a unified protocol for data interaction between cloud servers and terminals.

[0043] The terminal collaboration interface refers to the interface between the terminal software and hardware modules. Through this interface, you can effectively interact with the terminal, configure video encoding and rendering parameters, and obtain the real-time operating performance of the hardware.

[0044] The decoding protocols include video codec protocols such as H.264, H.265, and AV1, as well as the Profile and Level supported by the terminal under each codec protocol. Decoding performance refers to the highest supported decoding frame rate and single-frame decoding latency for a given video size under a specific decoding protocol.

[0045] The video sizes are defined as follows: 360p, 576p, 720p, 1080p, 2k, 4k. The video frame rates are defined as follows: 30fps, 40fps, 50fps, 60fps, 90fps, 120fps.

[0046] The decoding performance supported by the terminal is given in the form of a triple. The first element is an enumerated definition of the video resolution, the second element is an enumerated definition of the video frame rate, and the third element is the single-frame decoding latency under the video resolution and video frame rate. For example, the single-frame decoding latency of device A's H264 decoding at 720p@60fps is 10ms.

[0047] The cloud server determines the set of encoding functions that need to be enabled based on the game type and network conditions, and then determines the optimal encoding configuration for the current device based on the device type and encoding capabilities reported by the terminal.

[0048] The data structure requirements for terminal decoding capabilities are shown in Table 1:

[0049] Table 1. Data Structure Requirements for Terminal Decoding Capability

[0050]

[0051]

[0052] Based on the terminal's decoding capabilities, and combined with the game type and network communication information, the cloud server determines the optimal decoding protocol, decoding resolution, video frame rate, and other encoding and decoding configurations for the current device, as well as the number of video encoding reference frames and SVC enabling encoding and decoding strategies.

[0053] like Figure 1 The diagram shown illustrates the collaborative optimization connection process for video rendering:

[0054] 1. The cloud server initiates a request to the terminal through the START client (i.e., the cloud gaming client) to obtain the terminal's decoding capability information. The request protocol fields include the protocol version number and the specific encoding / decoding protocol query.

[0055] 2. When the terminal receives a capability information retrieval request, it returns a status flag (0 for success, a specific error code for failure), the supported protocol version number, and the terminal device capability information. If the terminal only supports some decoding protocols, it returns the supported decoding protocol information; if the terminal does not support any decoding protocols, it returns codecs=0; if the terminal capability information request fails, it returns the specific error code.

[0056] 3. After receiving the terminal's decoding capability information, the cloud server, in conjunction with the game type and network communication information, determines the optimal decoding protocol, decoding resolution, video frame rate, and other encoding and decoding configurations for the current terminal device, as well as the number of video encoding reference frames and SVC enabling encoding and decoding strategies.

[0057] 4. After receiving the optimal decoding configuration, the terminal decodes the video stream.

[0058] Among them, the START client is a cloud gaming client that is installed on the terminal.

[0059] The following is an example of a frequency coding collaborative optimization connection protocol:

[0060] 1. Video decoding capability request:

[0061]

[0062] 2. Video decoding capability response (supports all query decoding protocols):

[0063]

[0064]

[0065] 3. Video decoding capability response (only supports partial query decoding protocols):

[0066]

[0067] 4. Video decoding capability request (decoding protocol not supported):

[0068]

[0069] 5. Video decoding capability request (protocol request failed):

[0070]

[0071] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of an image processing system provided in an embodiment of this application. The image processing system includes a terminal 10 and a cloud server 20, etc.; the terminal 10 and the cloud server 20 are connected via a network, such as a wired or wireless network.

[0072] Terminal 10 can be used to display a graphical user interface (GUI). Terminal 10 is used to interact with the user through the GUI, such as downloading and installing a client, running a mini-program, or accessing a website. In this embodiment, terminal 10 can be a device that receives and decodes the bitstream transmitted from cloud server 20 to display game footage. Cloud server 20 transmits the encoded bitstream to terminal 10, which receives and decodes the bitstream, then plays video, displays game footage, etc., based on the decoded video data.

[0073] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0074] This application provides an image processing method in various embodiments. This method can be executed by an image processing system, such as by a terminal 10 or a cloud server 20, or by both a terminal 10 and a cloud server 20. This application uses the example of the image processing method being executed by both a cloud server 20 and a terminal 10 for illustration.

[0075] Please see Figure 3 Image processing methods include:

[0076] Step 210: When there are decoded cache images in terminal 10, a frame interpolation operation is performed on multiple consecutive first images to generate multiple consecutive second images. The frame interpolation operation includes inserting N intermediate images between any two adjacent first images. The intermediate images are generated based on the first images. The decoded cache images are images cached in the decoder of terminal 10. The value of N is determined based on the number of decoded cache images.

[0077] The cloud server 20 can detect whether the terminal 10 has a decoded cache image. For example, the cloud server 20 can obtain the decoding parameters of the decoder of the terminal 10 to determine whether the decoder has a decoded cache image.

[0078] When a decoding cache image exists on terminal 10, the decoding cache image will generate a decoding delay. For example, if there are multiple decoding cache images (i.e., buffer frames), each buffer frame requires a predetermined rendering time to decode. Therefore, after one image frame enters the decoder, three more images need to be input as buffer frames so that the one image frame can be output from the decoder. Thus, the single-frame decoding delay is three times the predetermined rendering time. The image data output from the decoder can be displayed on terminal 10. The predetermined rendering time includes the time for the decoder to decode the image and the time for the decoder to render the image.

[0079] At this time, when the cloud server 20 sends the encoded first image to the terminal 10, the terminal 10 can perform frame interpolation on multiple consecutive frames of the first image, thereby generating multiple consecutive frames of the second image after frame interpolation.

[0080] Frame interpolation can specifically involve inserting N intermediate images between any two adjacent first images. These intermediate images can be P-skip frames. Since the decoded image of a P-skip frame is basically the same as the previous frame, that is to say, the intermediate image can be generated based on the first image. The image data of the intermediate image and the first image are basically the same. Therefore, decoding the P-skip frame takes less time than decoding the first image.

[0081] The number of N can be determined based on the number of decoded buffer images. For example, the number of N is equal to the number of decoded buffer images. If the number of buffered frames is 3, then N = 3; if the number of buffered frames is 2, then N = 2.

[0082] Thus, by replacing the first image with the same number of intermediate images as the storage frames, since the decoding time of the intermediate images is much shorter than that of the first image, after inserting N intermediate images after the first image, the first image can be quickly output from the decoder, and the single-frame decoding delay is significantly shortened when N intermediate images are used as storage frames.

[0083] Step 220: Decode the second image.

[0084] After inserting N intermediate images after the first image, multiple consecutive frames of the second image can be obtained based on the first image and the intermediate images. The decoder of the terminal 10 decodes the second images sequentially according to the input order of the second images. Although the overall number of frames increases after frame insertion, the single-frame decoding delay of each image is shortened because the intermediate images can speed up the output of the first image and the decoding time of the intermediate images is shorter, thus improving the real-time performance of decoding.

[0085] Optionally, such as Figure 4 As shown, when decoding the second image, the image processing method further includes:

[0086] Step 230: Obtain the single-frame decoding delay and decoding output frame rate during decoding.

[0087] It is understandable that when the computing power of terminal 10 is insufficient, even if frame interpolation is performed, the single-frame decoding latency will not be reduced. Therefore, when terminal 10 decodes the second image, the single-frame decoding latency of the decoder can be obtained, so as to determine in real time whether the single-frame decoding latency after frame interpolation is compared with the single-frame decoding latency without frame interpolation.

[0088] Furthermore, when the computing power of terminal 10 is insufficient, even if frame interpolation is performed, the decoding output frame rate of the decoder is unstable. Therefore, by judging whether the decoding output frame rate after frame interpolation is stable in real time, it is possible to determine whether the computing power of terminal 10 is sufficient.

[0089] Frame interpolation will only continue on terminal 10 when there is sufficient computing power.

[0090] Optionally, such as Figure 4 As shown, step 210: When there is a decoded buffer image in terminal 10, perform frame interpolation on multiple consecutive frames of the first image to generate multiple consecutive frames of the second image, including:

[0091] Step 211: When the terminal 10 has a decoded buffer image, the single-frame decoding delay is less than the preset delay, and the difference between the decoded output frame rate and the preset input frame rate is less than the preset difference, the terminal 10 performs a frame interpolation operation on multiple consecutive first images before decoding to generate multiple consecutive second images.

[0092] When a decoding buffer image exists in terminal 10, frame interpolation can reduce the single-frame decoding latency. However, when no decoding buffer image exists in terminal 10, frame interpolation may increase the single-frame decoding latency of the first image.

[0093] Therefore, when terminal 10 has a decoded cached image, the single-frame decoding delay is less than the preset delay (i.e., the single-frame decoding delay after frame interpolation is less than the single-frame decoding delay without frame interpolation), and the difference between the decoded output frame rate and the preset input frame rate is less than the preset difference (i.e., the decoded output frame rate is basically consistent with the preset input frame rate, such as the difference between the decoded output frame rate and the preset input frame rate being less than 5% of the preset frame rate), it can be determined that terminal 10 has a decoded cached image and that terminal 10 has sufficient computing power. Therefore, cloud server 20 can send the encoded first image bitstream to terminal 10. Terminal 10 receives the bitstream sent by cloud server 20 and performs frame interpolation on multiple consecutive first images in the bitstream before decoding to generate multiple consecutive second images, thereby saving cloud server 20 resources and reducing costs.

[0094] Optionally, such as Figure 5 As shown, the image processing method also includes:

[0095] Step 240: Determine the rendering duration based on the preset input frame rate.

[0096] When determining whether terminal 10 has a decoded cached image, cloud server 20 can transmit multiple consecutive frames of the first image to terminal 10. The decoder can calculate the rendering time allocated to each frame of the first image based on a preset input frame rate. For example, rendering time = 1000 / preset input frame rate. The rendering time is the total time for the decoder to decode and render the first image.

[0097] Step 250: When the single-frame decoding delay is greater than the rendering time, it is determined that the terminal 10 has a decoded cached image.

[0098] When terminal 10 performs frame interpolation and decodes the multiple consecutive frames of the second image obtained after the frame interpolation, the decoder can obtain the single-frame decoding delay in real time. If the single-frame decoding delay is less than (or less than or equal to) the rendering time, it means that after the first image or intermediate image is input into the decoder, after decoding and rendering, terminal 10 does not have a decoding cache image. It can output from the decoder immediately without waiting for the frame decoding and rendering to be completed.

[0099] When the single-frame decoding delay is greater than the rendering time, it can be determined that there is a frame hoarding in terminal 10. After the first image or intermediate image is input into the decoder, it is necessary to wait for the hoarded frame to be decoded and rendered before it will be output, thus making the single-frame decoding delay greater than the rendering time.

[0100] The preset input frame rate can be determined in advance through different decoding parameters of the decoder. For example, the decoder includes different decoding parameters, including at least one of the following: input frame rate, encoding method, rendering method, and frame dropping mode. The input frame rate, encoding method, rendering method, and frame dropping mode all affect the single-frame decoding latency.

[0101] When performing decoding, the terminal 10 can select different decoding parameters to decode multiple consecutive frames of the first image, thereby obtaining the single-frame decoding delay corresponding to each decoding parameter.

[0102] The decoder can determine the decoding parameters corresponding to the minimum single-frame decoding delay as the target decoding parameters (that is, the input frame rate corresponding to the minimum single-frame decoding delay is the preset input frame rate), thereby ensuring that the terminal 10 decodes with the optimal decoding parameters.

[0103] Optionally, such as Figure 6 As shown, step 211, before decoding on terminal 10, involves interpolating multiple consecutive frames of the first image to generate multiple consecutive frames of the second image, including:

[0104] Step 212: Before decoding, the terminal 10 receives the bitstream sent by the cloud server 20. The bitstream includes multiple frames of the first image and encoding information. The encoding information includes the encoding method and resolution.

[0105] Before decoding, the terminal 10 can receive the bitstream sent by the cloud server 20. The bitstream contains multiple frames of first images and encoding information, which may include encoding method and resolution.

[0106] Step 213: Generate N intermediate images based on the encoding method and resolution.

[0107] After obtaining the encoding method and resolution, terminal 10 can generate N intermediate images based on the encoding method and resolution. For example, after the first image is input into the decoder, the image data of the intermediate images can be generated with reference to the first image. Terminal 10 can generate the next intermediate image based on the image data, encoding method, and resolution of the previous frame's first image or the previous frame's intermediate image, thereby generating N intermediate images.

[0108] Step 214: Insert N intermediate images between any two adjacent first images to generate multiple consecutive second images.

[0109] After generating N intermediate images, these N intermediate images can be inserted after the associated first image. Each first image corresponds to N intermediate images, and each first image is followed by N intermediate images, thus ensuring that N intermediate images are inserted between any two first images. The association of the N intermediate images with the first image means that the N intermediate images are generated with reference to the first image.

[0110] Optionally, such as Figure 7 As shown, step 221, which involves interpolating multiple consecutive frames of the first image before decoding on the terminal 10 to generate multiple consecutive frames of the second image, further includes:

[0111] Step 215: Determine the encoding method based on the gear position number and the entropy encoding flag.

[0112] There are different standards for video encoding and decoding, such as MPEG-2 and H.264 / AVC. This implementation method uses the H.264 / AVC standard as an example for explanation.

[0113] For the H.264 / AVC standard, encoding information may include the bit sequence number, entropy encoding flag, image width, and image height; the bit sequence number and entropy encoding flag have a preset association with the encoding method. Terminal 10 can determine the encoding method based on the bit sequence number and entropy encoding flag. For the H.264 / AVC standard, there are CAVLC and CABAC encoding methods; the bit sequence number and entropy encoding flag can accurately determine whether the encoding method is CAVLC or CABAC.

[0114] Step 216: Determine the resolution based on the image width and image height.

[0115] Terminal 10 can determine the resolution of the first image in each frame based on the image width and image height. For example, resolution = image width * image height.

[0116] After determining the encoding method and resolution, step 2212 can be executed to generate N intermediate images corresponding to the first image of each frame.

[0117] It is understandable that, for different video codec standards, the encoding method and resolution can be obtained through the encoding information corresponding to the video codec standard, and then used to generate N intermediate frames of images.

[0118] Optionally, such as Figure 8 As shown, step 210, when the terminal 10 has a decoded cached image, performs frame interpolation on multiple consecutive frames of the first image to generate multiple consecutive frames of the second image, and further includes:

[0119] Step 217: When the terminal 10 has a decoding cache image, the maximum number of reference frames is less than a preset number, and the maximum decoding cache image is less than a preset decoding cache image, a frame interpolation operation is performed on multiple consecutive first images before the terminal 10 performs decoding to generate multiple consecutive second images.

[0120] The encoding information may also include the maximum number of reference frames and the maximum number of decoded buffer images. The maximum number of reference frames is the number of images referenced during encoding of each frame, and the maximum number of decoded buffer images is the number of decoded buffer images. The maximum number of reference frames and the maximum number of decoded buffer images are the same. Since the image frames in the decoded buffer images are generally used as reference images for subsequently decoded images, the maximum number of reference frames and the maximum number of decoded buffer images are generally the same.

[0121] It is understandable that the larger the maximum number of reference frames and the maximum number of decoded buffer images, the greater the single-frame decoding latency caused by frame buffering. To further reduce single-frame decoding latency, frame interpolation can be performed on terminal 10 only when decoded buffer images exist in terminal 10, the maximum number of reference frames is less than a preset number, and the maximum number of decoded buffer images is less than a preset number of decoded buffer images, thereby generating multiple consecutive second images. For example, the preset number and preset number of decoded buffer images are both 2. In this way, only one frame is buffered for reference during subsequent image encoding, maximizing the reduction of single-frame decoding latency while ensuring that the decoder has decoded buffer images for reference in decoding.

[0122] Optionally, such as Figure 9 As shown, step 210, when the terminal 10 has a decoded cached image, performs frame interpolation on multiple consecutive frames of the first image to generate multiple consecutive frames of the second image, and further includes:

[0123] Step 218: When there is a decoded cached image on terminal 10 and the single-frame decoding delay is greater than the preset delay or the difference between the decoded output frame rate and the preset input frame rate is greater than the preset difference, a frame interpolation operation is performed on the cloud server 20 that is connected to terminal 10 to generate multiple consecutive frames of the second image.

[0124] When there are cached images in the terminal 10, frame interpolation is required. However, if the single-frame decoding delay is greater than (or greater than or equal to) the preset delay, or the difference between the decoded output frame rate and the preset input frame rate is greater than (or greater than or equal to) the preset difference, it can be determined that the single-frame decoding delay has not been reduced after the frame interpolation operation in the terminal 10, or the single-frame decoding delay has been reduced but the decoded output frame rate is very unstable. Therefore, it can be determined that the computing power of the terminal 10 is insufficient. In this case, the powerful computing power of the cloud can be used to perform the frame interpolation operation to reduce the computing power consumption of the terminal 10, thereby reducing the power consumption of the terminal 10.

[0125] Optionally, such as Figure 10 As shown, step 218: Perform frame interpolation on the cloud server 20, which is communicatively connected to the terminal 10, to generate multiple consecutive frames of the second image, including:

[0126] Step 2181: Encode multiple consecutive frames of the first image to obtain multiple consecutive frames of the third image after encoding.

[0127] During encoding, multiple consecutive frames of the first image can be encoded first to obtain multiple consecutive frames of the third image after encoding.

[0128] Step 2182: Encode each frame of the first image N times to generate N intermediate images corresponding to each frame of the first image, so that the third image, the first image and the N intermediate images correspond to each other.

[0129] Then, the cloud server 20 can encode the first image of each frame N times again, thereby generating N corresponding intermediate images for each first image, so that the first image corresponds to the third image and also corresponds to the N intermediate images. That is to say, the third image, the first image and the N intermediate images correspond to each other.

[0130] Step 2183: Insert N target intermediate images between any two adjacent third images to generate multiple consecutive second images. The N target intermediate images are the N intermediate images corresponding to the previous frame in any two adjacent third images.

[0131] Finally, the cloud server 20 can insert N target intermediate images between two adjacent third image frames. The target intermediate images are the N intermediate images corresponding to the previous third image in the two adjacent third image frames, thus ensuring that there are N intermediate images between any two adjacent third image frames, achieving the frame insertion operation. After the frame insertion operation, the sequence numbers of the multi-frame second images generated after frame insertion can be edited to ensure that the sequence numbers of consecutive second images are continuous, facilitating subsequent decoding by the decoder.

[0132] In other words, the cloud server 20 encodes each frame of the first image N+1 times. Since the encoded third image and the N intermediate images are both generated based on the first image, the N intermediate images can be encoded with reference to the third image corresponding to the first image, thus making the amount of data after encoding the intermediate images smaller.

[0133] Therefore, by repeatedly encoding the first image through the cloud server 20, N frames of target intermediate images are generated, thereby realizing frame interpolation operation, reducing the computing power consumption of the terminal 10, as well as the single-frame decoding latency of the terminal 10 during decoding.

[0134] Optionally, such as Figure 11 As shown, the image processing method also includes:

[0135] Step 260: Determine the interpolation input frame rate based on the preset input frame rate and the value of N. The preset input frame rate is the input frame rate when the decoder decodes multiple consecutive frames of the first image.

[0136] After performing frame interpolation, terminal 10 can determine the interpolation input frame rate based on the preset input frame rate and the value of N. For example, when terminal 10 decodes multiple consecutive first images, the preset input frame rate is 30 frames / second. After interpolating N intermediate images into each first image (e.g., N is 1), the decoder's input frame rate will change. The interpolated input frame rate after interpolation = preset input frame rate * (N + 1), which is 60 frames / second.

[0137] Step 270: Determine the rendering duration based on the interpolation input frame rate.

[0138] Once the interpolation input frame rate is determined, the rendering time that the decoder allocates for the second image in each frame after interpolation can be determined. Rendering time = 1000 / interpolation input frame rate.

[0139] Step 280: Determine the maximum number of interpolated frames based on the rendering duration and preset delay.

[0140] Then, the terminal 10 can determine the maximum number of interpolated frames based on the rendering duration and the preset delay. The preset delay is determined based on the single-frame decoding delay when the decoder decodes multiple consecutive frames of the first image.

[0141] It can be understood that the number of interpolated frames N should satisfy N * rendering time < preset latency, that is, the decoding latency of a single frame after interpolation should be less than the decoding latency of a single frame before interpolation. In other words, 1000N / (preset input frame rate * (N+1)) < preset latency, thus the maximum number of interpolated frames can be calculated.

[0142] Therefore, when determining the value of N based on the decoded buffer images, N can be made less than the maximum number of interpolated frames, ensuring that the decoding latency of a single frame after interpolation is less than that before interpolation. Alternatively, N can be made equal to the number of decoded buffer images, allowing the decoder to minimize the number of decoded buffer images while ensuring sufficient decoded buffer images as reference frames, thereby reducing the single-frame decoding latency.

[0143] Optionally, such as Figure 5 As shown, step 220: Decoding the second image includes:

[0144] Step 221: Input one frame of the second image to the decoder every rendering time.

[0145] The rendering time is generally longer than the total decoding and rendering time required to decode one frame of the second image. After the decoder has finished decoding and rendering one frame of the second image, in order to ensure a stable decoding output frame rate, even if the decoder finishes decoding and rendering one frame before the rendering time ends, it will wait until the rendering time ends before inputting the next frame of the second image for decoding.

[0146] Step 222: After the decoder has decoded one frame of the second image, decode the next frame of the second image.

[0147] If the rendering time exceeds the limit before the second frame of a certain image is fully decoded and rendered, the next second frame of the image is input into the decoding buffer. The process continues until the second frame of the current image is fully decoded and rendered before the next second frame of the image is decoded, thus ensuring that each second frame of the image is fully decoded and rendered.

[0148] In general, only the second image corresponding to the first image in the second image may exceed the rendering time. However, due to the existence of decoding cache images, even if the decoding and rendering time of the second image corresponding to the first image exceeds the rendering time, the decoding and rendering time of the second image corresponding to the intermediate images is shorter (less than the rendering time). Therefore, the second images corresponding to the intermediate images of N frames in the decoding cache image can save some rendering time, which is used for decoding and rendering of the second image corresponding to the first image. This makes the total decoding and rendering time of the second image corresponding to one frame of the first image and the second images corresponding to N frames of intermediate images less than (N+1)*rendering time. This ensures that the decoding latency of a single frame is reduced while maintaining the stability of the decoding output frame rate.

[0149] Optionally, before decoding, the method further includes:

[0150] The cloud gaming client sends a request to the terminal to obtain information about the terminal's encoding and decoding capabilities.

[0151] Upload encoding / decoding capability information as requested;

[0152] Based on the encoding / decoding capability information, the type of game running on the terminal, and network communication information, determine the target encoding / decoding configuration and target encoding / decoding strategy;

[0153] Send the target encoding / decoding configuration and target encoding / decoding strategy to the terminal via the cloud gaming client;

[0154] Receive target encoding / decoding configuration and target encoding / decoding strategy.

[0155] For example, please combine Figure 1 The flowchart shown illustrates that the cloud (cloud server) initiates a request to the terminal (smart terminal) to obtain the encoding and decoding capability information through a cloud gaming client (such as the START client).

[0156] The request protocol field may include information such as the protocol version number and specific codec protocol query. When the terminal receives a capability information retrieval request, it uploads the codec capability information to the cloud gaming client, which then returns the terminal's codec capability information to the cloud. This codec capability information may include a status flag, supported protocol version numbers, and terminal device capability information, which may include terminal encoding capability information and terminal decoding capability information.

[0157] The capability information acquisition request may include a decoding capability request and / or an encoding capability request. After receiving the encoding and decoding capability information, the cloud can determine the target encoding and decoding configuration and target encoding and decoding strategy based on the encoding and decoding capability information, game type, and network communication information. The target encoding and decoding configuration may include the optimal decoding protocol, decoding resolution, video frame rate, etc. for the current terminal device, and the target encoding and decoding strategy may include the number of video encoding reference frames, SVC enabling, etc.

[0158] The cloud can determine the set of encoding functions to be enabled based on the game type and network conditions, and then determine the optimal encoding configuration for the current terminal based on the device type and encoding capability information reported by the terminal. The cloud can also determine the optimal encoding and decoding configuration for the current terminal, such as decoding protocol, decoding resolution, video frame rate, video encoding reference frame number, and SVC enablement, based on the terminal's decoding capability information and in combination with the game type and network conditions.

[0159] Then, the cloud sends the target encoding / decoding configuration and target encoding / decoding strategy to the terminal. After receiving the target encoding / decoding configuration and target encoding / decoding strategy, the terminal decodes the second image based on the target encoding / decoding configuration and target encoding / decoding strategy.

[0160] For a better illustration of the image processing method provided in the embodiments of this application, please refer to... Figure 13 The image processing method provided in this application embodiment can be summarized into the following steps:

[0161] Step 1210: Obtain the single-frame decoding delay and decoding output frame rate during decoding to determine whether the computing power of terminal 10 is sufficient;

[0162] Step 1220: When the computing power of the terminal 10 is sufficient, encode multiple consecutive frames of the first image and send the encoded bitstream to the terminal 10.

[0163] Step 1230: Receive the bitstream sent by the cloud server 20, and perform frame interpolation on multiple consecutive frames of the first image in the bitstream;

[0164] Step 1240: When the computing power of terminal 10 is insufficient, the cloud server 20 performs frame interpolation on multiple consecutive frames of the first image and sends the encoded bitstream to terminal 10.

[0165] Step 1250: Decode the consecutive frames of the second image obtained after the frame interpolation operation.

[0166] In this context, step 1210 can be referred to the descriptions of steps 230 and 211, steps 1220 and 1230 can be referred to the description of step 211, step 1240 can be referred to the description of step 218, and step 1250 can be referred to the description of step 220.

[0167] Another embodiment of this application will be described using an image processing method executed by terminal 10 as an example.

[0168] Please see Figure 14 Image processing methods include:

[0169] Step 310: When there is a decoding cache image in the terminal 10, before the terminal 10 performs decoding, a frame interpolation operation is performed on multiple consecutive first images to generate multiple consecutive second images. The frame interpolation operation includes inserting N intermediate images between any two adjacent first images. The intermediate images are generated based on the first images. The decoding cache images are images cached in the decoder of the terminal 10. The value of N is determined based on the number of decoding cache images.

[0170] Optionally, terminal 10 can detect whether the decoder has a decoding cache image. For example, terminal 10 can obtain the decoder's decoding parameters (such as the maximum decoding cache image) to determine whether the decoder has a decoding cache image. When the maximum decoding cache image is greater than 0, it can be determined that the decoder has a decoding cache image; when the maximum decoding cache image is equal to 0, it is determined that the decoder does not have a decoding cache image.

[0171] Terminal 10 receives a series of first images after encoding from cloud server 20. If terminal 10 has a decoded cache image, before terminal 10 performs decoding, terminal 10 can perform frame interpolation on the series of first images to generate a series of second images after frame interpolation.

[0172] For details on the frame interpolation operation, please refer to step 210, which will not be repeated here.

[0173] Thus, by using intermediate images instead of the first image as storage frames, since the decoding time of the intermediate images is much shorter than that of the first image, after inserting N intermediate images after the first image, the first image can be quickly output from the decoder, and the single-frame decoding latency is significantly shortened when using N intermediate images as storage frames.

[0174] Step 320: Decode the second image.

[0175] After inserting N intermediate images after the first image, multiple consecutive frames of the second image can be obtained based on the first image and the intermediate images. The decoder of the terminal 10 decodes the second images sequentially according to the input order of the second images. Although the overall number of frames increases after frame insertion, the single-frame decoding delay of each image is shortened because the intermediate images can speed up the output of the first image and the decoding time of the intermediate images is shorter, thus improving the real-time performance of decoding.

[0176] Optionally, such as Figure 15As shown, when decoding the second image, the image processing method further includes:

[0177] Step 330: Obtain the single-frame decoding delay and decoding output frame rate during decoding.

[0178] For a detailed description of step 330, please refer to step 230, which will not be repeated here.

[0179] Step 310: When the terminal 10 has a decoding buffer image, before the terminal 10 performs decoding, a frame interpolation operation is performed on multiple consecutive frames of the first image to generate multiple consecutive frames of the second image, including:

[0180] Step 311: When the terminal 10 has a decoded buffer image, the single-frame decoding delay is less than the preset delay, and the difference between the decoded output frame rate and the preset input frame rate is less than the preset difference, the terminal 10 performs a frame interpolation operation on multiple consecutive first images before decoding to generate multiple consecutive second images.

[0181] For a detailed description of step 311, please refer to step 211, which will not be repeated here.

[0182] Optionally, such as Figure 16 As shown, step 310 involves performing frame interpolation on multiple consecutive frames of the first image to generate multiple consecutive frames of the second image, including:

[0183] Step 312: Receive the bitstream sent by the cloud server 20. The bitstream includes multiple frames of the first image and encoding information. The encoding information includes the encoding method and resolution.

[0184] Before decoding, the terminal 10 can receive the bitstream sent by the cloud server 20. The bitstream contains multiple frames of first images and encoding information, which may include encoding method and resolution.

[0185] Step 313: Generate N intermediate images based on the encoding method and resolution.

[0186] For a detailed description of step 313, please refer to step 213, which will not be repeated here.

[0187] Step 314: Insert N intermediate images between any two adjacent first images to generate multiple consecutive second images.

[0188] For a detailed description of step 314, please refer to step 214, which will not be repeated here.

[0189] Optionally, such as Figure 17 As shown, step 310, when the terminal 10 has a decoded cached image, performs frame interpolation on multiple consecutive frames of the first image to generate multiple consecutive frames of the second image, and further includes:

[0190] Step 315: When the terminal 10 has a decoding cache image, the maximum number of reference frames is less than a preset number, and the maximum decoding cache image is less than a preset decoding cache image, a frame interpolation operation is performed on multiple consecutive first images before the terminal 10 performs decoding to generate multiple consecutive second images.

[0191] For a detailed description of step 315, please refer to step 217, which will not be repeated here.

[0192] Optionally, before decoding, the method further includes:

[0193] Obtain information about the terminal's encoding and decoding capabilities;

[0194] Upload encoding / decoding capability information to the cloud server via the client;

[0195] Obtain the target codec configuration and target codec strategy from the cloud server. The target codec configuration and target codec strategy are determined by the cloud server based on codec capability information, the type of game running on the terminal, and network communication information.

[0196] Obtain the first image of a series of consecutive frames generated by the cloud server after encoding the original image according to the target encoding / decoding configuration and target encoding / decoding strategy.

[0197] For example, please combine Figure 1 The flowchart shown illustrates that the cloud (cloud server) initiates a request to the terminal (smart terminal) to obtain the encoding and decoding capability information through a cloud gaming client (such as the START client).

[0198] The request protocol field may include information such as the protocol version number and specific codec protocol query. When the terminal receives a capability information retrieval request, it uploads the codec capability information to the cloud gaming client, which then returns the terminal's codec capability information to the cloud. This codec capability information may include a status flag, supported protocol version numbers, and terminal device capability information, which may include terminal encoding capability information and terminal decoding capability information.

[0199] The capability information acquisition request can include decoding capability requests and / or encoding capability requests. After receiving the encoding and decoding capability information, the cloud can determine the target encoding and decoding configuration and target encoding and decoding strategy based on the encoding and decoding capability information, game type, and network communication information. The target encoding and decoding configuration can include the optimal decoding protocol, decoding resolution, video frame rate, etc. for the current terminal device, and the target encoding and decoding strategy can include the number of video encoding reference frames, SVC enabling, etc.

[0200] The cloud can determine the set of encoding functions to be enabled based on the game type and network conditions, and then determine the optimal encoding configuration for the current terminal based on the device type and encoding capability information reported by the terminal. The cloud can also determine the optimal encoding and decoding configuration for the current terminal, such as decoding protocol, decoding resolution, video frame rate, video encoding reference frame number, and SVC enablement, based on the terminal's decoding capability information and in combination with the game type and network conditions.

[0201] Then, the cloud sends the target encoding / decoding configuration and target encoding / decoding strategy to the terminal. After receiving the target encoding / decoding configuration and target encoding / decoding strategy, the terminal decodes the second image based on the target encoding / decoding configuration and target encoding / decoding strategy.

[0202] Another embodiment of this application is illustrated by taking the image processing method executed by cloud server 20 as an example.

[0203] Please see Figure 18 Image processing methods include:

[0204] Step 410: When it is detected that there is a decoded cache image on the terminal 10 connected to the cloud server 20, a frame interpolation operation is performed on multiple consecutive first images during the encoding process of the cloud server 20 to generate multiple consecutive second images. The frame interpolation operation includes inserting N intermediate images between any two adjacent first images. The intermediate images are generated based on the first images. The decoded cache images are images cached in the decoder of the terminal 10. The value of N is determined according to the number of decoded cache images.

[0205] Optionally, the cloud server 20 can detect whether the decoder of the terminal 10 connected to it has a decoding cache image. For example, the terminal 10 can obtain the decoding parameters of the decoder (such as the maximum decoding cache image) to determine whether the decoder has a decoding cache image. When the maximum decoding cache image is greater than 0, it can be determined that the decoder has a decoding cache image; when the maximum decoding cache image is equal to 0, it does not have a decoding cache image.

[0206] When a terminal 10 connected to the cloud server 20 detects that there is a decoded cached image, the cloud server 20 can perform frame interpolation on multiple consecutive frames of the first image during encoding, thereby generating multiple consecutive frames of the second image after frame interpolation.

[0207] For details on the frame interpolation operation, please refer to step 210, which will not be repeated here.

[0208] Thus, by using intermediate images instead of the first image as storage frames, since the decoding time of the intermediate images is much shorter than that of the first image, after inserting N intermediate images after the first image, the first image can be quickly output from the decoder, and the single-frame decoding latency is significantly shortened when using N intermediate images as storage frames.

[0209] Step 420: The encoded consecutive frames of the second image are sent to the terminal 10 so that the terminal 10 can decode the second image.

[0210] The cloud server 20 sends the encoded consecutive frames of the second image to the terminal 10. The decoder of the terminal 10 decodes the second image sequentially according to the input order of the second image. Although the overall number of frames increases after frame interpolation, the single-frame decoding latency of each image is shortened because the intermediate image can speed up the output of the first image and the decoding time of the intermediate image is shorter, thus improving the real-time performance of decoding.

[0211] Optionally, such as Figure 19 As shown, step 410: performing frame interpolation on the cloud server 20, which is communicatively connected to the terminal 10, to generate multiple consecutive frames of the second image, further includes:

[0212] Step 411: Encode multiple consecutive frames of the first image to obtain multiple consecutive frames of the third image after encoding.

[0213] For a detailed description of step 411, please refer to step 2181, which will not be repeated here.

[0214] Step 412: Encode each frame of the first image N times to generate N intermediate images corresponding to each frame of the first image, so that the third image, the first image and the N intermediate images correspond to each other.

[0215] For a detailed description of step 412, please refer to step 2182, which will not be repeated here.

[0216] Step 413: Insert N target intermediate images between any two adjacent third images to generate multiple consecutive second images. The N target intermediate images are the N intermediate images corresponding to any frame in any two adjacent third images to generate multiple consecutive second images.

[0217] For a detailed description of step 413, please refer to step 2183, which will not be repeated here.

[0218] Optionally, such as Figure 20 As shown, the image processing method also includes:

[0219] 430. After the decoder finishes decoding the second image, obtain the single-frame decoding delay of the decoder.

[0220] After the decoder finishes decoding each frame of the second image, the cloud server 20 can obtain the single-frame decoding latency corresponding to that frame of the second image. Here, "the decoder finishes decoding one frame of the second image" refers to the time from when the second image enters the decoder to when it is output from the decoder.

[0221] 440. If the single-frame decoding delay is greater than the preset delay, the frame interpolation operation is stopped. The preset delay is determined based on the single-frame decoding delay when the decoder decodes multiple consecutive first images.

[0222] It is understandable that when the computing power of terminal 10 is severely insufficient or the frame rate after frame interpolation exceeds the maximum decoding frame rate of terminal 10, it may be possible that even if frame interpolation is performed, the single-frame decoding latency cannot be reduced. Therefore, if, after frame interpolation, the single-frame decoding latency of the decoder of terminal 10 after decoding a frame of the second image is greater than (or greater than or equal to) the preset latency, it means that the single-frame decoding latency after frame interpolation is greater than the single-frame decoding latency before frame interpolation.

[0223] Therefore, the frame interpolation operation needs to be stopped immediately, and the third image after multiple consecutive frames of encoding is directly transmitted to terminal 10 for decoding, thereby reducing the single-frame decoding latency.

[0224] Of course, the single-frame decoding latency after frame interpolation may sometimes fluctuate due to insufficient terminal computing power. Therefore, when determining whether the single-frame decoding latency is greater than the preset latency, the average value of the single-frame decoding latency corresponding to multiple consecutive decoded second images can be obtained. The single-frame decoding latency is determined by whether the average value of the multiple single-frame decoding latency is greater than the preset latency. If the average value of the multiple single-frame decoding latency is greater than the preset latency, the single-frame decoding latency is determined to be greater than the preset latency. If the average value of the multiple single-frame decoding latency is less than the preset latency, the single-frame decoding latency is determined to be less than the preset latency. This reduces the probability of misjudgment caused by fluctuations in the single-frame decoding latency.

[0225] Optionally, before decoding, the method further includes:

[0226] The cloud gaming client sends a request to the terminal to obtain information about the terminal's encoding and decoding capabilities.

[0227] Receive encoding / decoding capability information uploaded by the terminal;

[0228] The target codec configuration and target codec strategy are determined based on codec capability information, the type of game running on the terminal, and network communication information.

[0229] Send the target encoding / decoding configuration and target encoding / decoding strategy to the terminal via the cloud gaming client;

[0230] Encode multiple consecutive frames of the first image to obtain multiple consecutive frames of the encoded third image, including:

[0231] Based on the target encoding / decoding configuration and target encoding / decoding strategy, multiple consecutive frames of the first image are encoded to obtain multiple consecutive frames of the third image.

[0232] Decoding the second image includes:

[0233] The second image is decoded according to the target encoding / decoding configuration and target encoding / decoding strategy.

[0234] For example, please combine Figure 1 The flowchart shown illustrates that the cloud (cloud server) initiates a request to the terminal (smart terminal) to obtain the encoding and decoding capability information through a cloud gaming client (such as the START client).

[0235] The request protocol field may include information such as the protocol version number and specific codec protocol query. When the terminal receives a capability information retrieval request, it uploads the codec capability information to the cloud gaming client, which then returns the terminal's codec capability information to the cloud. This codec capability information may include a status flag, supported protocol version numbers, and terminal device capability information, which may include terminal encoding capability information and terminal decoding capability information.

[0236] The capability information acquisition request can include decoding capability requests and / or encoding capability requests. After receiving the encoding and decoding capability information, the cloud can determine the target encoding and decoding configuration and target encoding and decoding strategy based on the encoding and decoding capability information, game type, and network communication information. The target encoding and decoding configuration can include the optimal decoding protocol, decoding resolution, video frame rate, etc. for the current terminal device, and the target encoding and decoding strategy can include the number of video encoding reference frames, SVC enabling, etc.

[0237] The cloud can determine the set of encoding functions to be enabled based on the game type and network conditions, and then determine the optimal encoding configuration for the current terminal based on the device type and encoding capability information reported by the terminal. The cloud can also determine the optimal encoding and decoding configuration for the current terminal, such as decoding protocol, decoding resolution, video frame rate, video encoding reference frame number, and SVC enablement, based on the terminal's decoding capability information and in combination with the game type and network conditions.

[0238] Then, the cloud can encode multiple consecutive frames of the first image based on the target encoding and decoding configuration and the target encoding and decoding strategy to obtain multiple consecutive frames of the third image and N intermediate images corresponding to each frame of the first image. After performing frame interpolation on the multiple consecutive frames of the third image, multiple consecutive frames of the second image can be obtained. The cloud transmits the bitstream of the multiple consecutive frames of the second image to the terminal through the cloud gaming client.

[0239] The cloud can send target encoding / decoding configuration and target encoding / decoding strategy to the terminal. After receiving the target encoding / decoding configuration and target encoding / decoding strategy, the terminal decodes the second image based on the target encoding / decoding configuration and target encoding / decoding strategy.

[0240] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0241] This application embodiment performs frame interpolation on multiple consecutive first images, thereby generating multiple consecutive second images by inserting N intermediate images generated from the first images between two adjacent first images, and then decoding the second images. During decoding, the number of frames stored in the decoder is fixed. After one first image is input into the decoder, compared to the existing scheme that inputs multiple adjacent first images as decoding buffer images, inputting N adjacent intermediate images as decoding buffer images results in a shorter decoding time for the intermediate images, since they are generated from the first images. This allows the first images to be output from the decoder quickly, reducing single-frame decoding latency.

[0242] In addition, this application embodiment can also determine whether the frame interpolation operation is performed on the cloud server 20 or the terminal 10 by judging whether the computing power of the terminal 10 is sufficient. Under the premise of ensuring that the single frame decoding latency can be reduced, the computing power of the terminal 10 can be fully utilized, and the computing cost of the cloud server 20 can be reduced.

[0243] To facilitate better implementation of the image processing method of this application embodiment, this application embodiment also provides an image processing apparatus. Please refer to... Figure 21 , Figure 21 This is a schematic diagram of the structure of an image processing apparatus 1000 provided in an embodiment of this application. The image processing apparatus 1000 may include:

[0244] The first frame insertion module 1010 is used to perform frame insertion operations on multiple consecutive first images when there are decoded cache images in the terminal 10, so as to generate multiple consecutive second images. The frame insertion operation includes inserting N intermediate images between any two adjacent first images. The intermediate images are generated based on the first images. The decoded cache images are images cached in the decoder of the terminal 10. The value of N is determined according to the number of decoded cache images.

[0245] The first decoding module 1020 is used to decode the second image.

[0246] The first acquisition module 1030 is used to acquire the single-frame decoding delay and decoding output frame rate during decoding.

[0247] Optionally, the first frame interpolation module 1010 may also be used for:

[0248] When there is a decoded buffer image in terminal 10, the single-frame decoding delay is less than the preset delay, and the difference between the decoded output frame rate and the preset input frame rate is less than the preset difference, the terminal 10 performs a frame interpolation operation on multiple consecutive first images before decoding to generate multiple consecutive second images.

[0249] The image processing apparatus 1000 also includes:

[0250] The first determining module 1040 is used to determine the rendering duration based on the preset input frame rate.

[0251] The second determining module 1050 is used to determine that the terminal 10 has a decoded cached image when the single-frame decoding delay is greater than the rendering time.

[0252] Optionally, the first frame interpolation module 1010 may also be used for:

[0253] Before decoding, the terminal 10 receives the bitstream sent by the cloud server 20. The bitstream includes multiple frames of the first image and encoding information, including the encoding method and resolution.

[0254] Based on the encoding method and resolution, generate N intermediate images;

[0255] Insert N intermediate images between any two adjacent first images to generate multiple consecutive second images.

[0256] Optionally, the encoding information may include the file number, entropy encoding flag, image width, and image height. Specifically, the first frame interpolation module 1010 may also be used for:

[0257] The encoding method is determined based on the gear position number and the entropy encoding flag.

[0258] The resolution is determined based on the image width and image height.

[0259] Optionally, the first frame interpolation module 1010 may also be used for:

[0260] When terminal 10 has a decoded cache image, the maximum number of reference frames is less than a preset number, and the maximum decoded cache image is less than a preset decoded cache image, a frame interpolation operation is performed on multiple consecutive first images before terminal 10 performs decoding to generate multiple consecutive second images.

[0261] Optionally, the first frame interpolation module 1010 may also be used for:

[0262] When there is a decoded cached image in terminal 10, and the single-frame decoding delay is greater than the preset delay or the difference between the decoded output frame rate and the preset input frame rate is greater than the preset difference, the cloud server 20, which is connected to terminal 10, performs frame interpolation to generate multiple consecutive frames of the second image.

[0263] Optionally, the first frame interpolation module 1010 may also be used for:

[0264] Encode multiple consecutive frames of the first image to obtain multiple consecutive frames of the third image;

[0265] The first image of each frame is encoded N times to generate N intermediate images corresponding to the first image of each frame, so that the third image, the first image and the N intermediate images correspond to each other.

[0266] N target intermediate images are inserted between any two adjacent third images to generate multiple consecutive second images. The N target intermediate images are the N intermediate images corresponding to the previous frame in any two adjacent third images.

[0267] The image processing apparatus 1000 also includes:

[0268] The third determining module 1060 is used to determine the interpolation input frame rate based on the preset input frame rate and the value of N. The preset input frame rate is the input frame rate when the decoder decodes multiple consecutive frames of the first image.

[0269] The fourth determining module 1070 is used to determine the rendering duration based on the interpolation input frame rate.

[0270] The fifth determining module 1080 is used to determine the maximum number of interpolated frames based on the rendering duration and preset delay.

[0271] Optionally, the first decoding module 1020 may also be used for:

[0272] A second image is input to the decoder every rendering interval;

[0273] After the decoder finishes decoding one frame of the second image, it decodes the next frame of the second image.

[0274] To facilitate better implementation of the image processing method of the embodiments of this application, another embodiment of this application also provides an image processing apparatus 2000. Please refer to... Figure 22 , Figure 22 This is a schematic diagram of the structure of an image processing apparatus 2000 provided in an embodiment of this application. The image processing apparatus 2000 may include:

[0275] The second frame interpolation module 2010 is used to perform frame interpolation operations on multiple consecutive first images before decoding on the terminal 10 when the terminal 10 has a decoding cache image, so as to generate multiple consecutive second images. The frame interpolation operation includes inserting N intermediate images between any two adjacent first images. The intermediate images are generated based on the first images. The decoding cache images are images cached in the decoder of the terminal 10. The value of N is determined according to the number of decoding cache images.

[0276] The second decoding module 2020 decodes the second image.

[0277] The second acquisition module 2030 acquires the single-frame decoding delay and decoding output frame rate during decoding.

[0278] Optionally, the second frame interpolation module 2010 can also be used for:

[0279] When there is a decoded buffer image in terminal 10, the single-frame decoding delay is less than the preset delay, and the difference between the decoded output frame rate and the preset input frame rate is less than the preset difference, the terminal 10 performs a frame interpolation operation on multiple consecutive first images before decoding to generate multiple consecutive second images.

[0280] Optionally, the second frame interpolation module 2010 can also be used for:

[0281] Receive the bitstream sent by cloud server 20;

[0282] Based on the encoding method and resolution, generate N intermediate images;

[0283] Insert N intermediate images between any two adjacent first images to generate multiple consecutive second images.

[0284] Optionally, the second frame interpolation module 2010 can also be used for:

[0285] When terminal 10 has a decoded cache image, the maximum number of reference frames is less than a preset number, and the maximum decoded cache image is less than a preset decoded cache image, a frame interpolation operation is performed on multiple consecutive first images before terminal 10 performs decoding to generate multiple consecutive second images.

[0286] To facilitate better implementation of the image processing method of the embodiments of this application, another embodiment of this application also provides an image processing apparatus 3000. Please refer to... Figure 23 , Figure 23 This is a schematic diagram of the structure of an image processing apparatus 3000 provided in an embodiment of this application. The image processing apparatus 3000 may include:

[0287] The third frame interpolation module 3010 is used to perform frame interpolation operations on multiple consecutive first images during encoding by the cloud server 20 when a decoded cache image is detected on the terminal 10 connected to the cloud server 20, so as to generate multiple consecutive second images. The frame interpolation operation includes inserting N intermediate images between any two adjacent first images. The intermediate images are generated based on the first images. The decoded cache images are images cached in the decoder of the terminal 10. The value of N is determined according to the number of decoded cache images.

[0288] The transmission module 3020 decodes the second image.

[0289] Optionally, the third frame interpolation module 3010 can also be used for:

[0290] Encode multiple consecutive frames of the first image to obtain multiple consecutive frames of the third image;

[0291] The first image of each frame is encoded N times to generate N intermediate images corresponding to the first image of each frame, so that the third image, the first image and the N intermediate images correspond to each other.

[0292] N target intermediate images are inserted between any two adjacent third images to generate multiple consecutive second images. The N target intermediate images are the N intermediate images corresponding to any frame in any two adjacent third images to generate multiple consecutive second images.

[0293] The image processing apparatus 3000 also includes:

[0294] The third acquisition module 3030 is used to acquire the single-frame decoding delay of the decoder after the decoder has finished decoding the second image.

[0295] The stop module 3040 is used to stop the frame interpolation operation when the single-frame decoding delay is greater than the preset delay. The preset delay is determined based on the single-frame decoding delay when the decoder decodes multiple consecutive frames of the first image.

[0296] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0297] The image processing device can be integrated into a terminal 10 and / or a cloud server 20 that has storage and a processor and thus computing power, or the image processing device can be the terminal 10 and / or the cloud server 20.

[0298] Optionally, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0299] Figure 24 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device may be... Figure 2 The terminal 10 or cloud server 20 shown. Figure 3As shown, the computer device 4000 may include: a communication interface 4010, a memory 4020, a processor 4030, and a communication bus 4040. The communication interface 4010, memory 4020, and processor 4030 communicate with each other via the communication bus 4040. The communication interface 4010 is used for data communication between the image processing device 1000 and external devices. The memory 4020 can be used to store software programs and modules, and the processor 4030 runs the software programs and modules stored in the memory 4020, such as the software programs for corresponding operations in the aforementioned method embodiments.

[0300] Optionally, the processor 4030 may call the software program and modules stored in the memory 4020 to perform the following operations: when there are decoded cache images in the terminal 10, perform frame interpolation operation on multiple consecutive first images to generate multiple consecutive second images. The frame interpolation operation includes inserting N intermediate images between any two adjacent first images. The intermediate images are generated based on the first images. The decoded cache images are images cached in the decoder of the terminal 10. The value of N is determined according to the number of decoded cache images. Decode the second images.

[0301] This application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a computer device, and the computer program causes the computer device to execute the corresponding processes in the image processing methods of the embodiments of this application; for the sake of brevity, further details are omitted here.

[0302] This application also provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding process in the image processing method of this application embodiment. For brevity, further details are omitted here.

[0303] This application also provides a computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding process in the image processing method of this application embodiment. For brevity, further details are omitted here.

[0304] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0305] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0306] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0307] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0308] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0309] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0310] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0311] In addition, the functional modules in the embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0312] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer or cloud server 20) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0313] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: When a decoded cache image exists on the terminal, a frame interpolation operation is performed on multiple consecutive frames of the first image to generate multiple consecutive frames of the second image. The frame interpolation operation includes inserting N intermediate images between any two adjacent frames of the first image. These intermediate images are generated based on the first images. The decoded cache image is an image cached in the terminal's decoder, and the value of N is determined based on the number of decoded cache images. The second image is decoded, and the single-frame decoding delay and decoding output frame rate of the decoder are obtained. The step of performing frame interpolation on multiple consecutive first images to generate multiple consecutive second images when the terminal has a decoding cache image includes: when the terminal has a decoding cache image, the single-frame decoding delay is less than a preset delay, and the difference between the decoding output frame rate and the preset input frame rate is less than a preset difference, performing frame interpolation on multiple consecutive first images before the terminal performs decoding to generate multiple consecutive second images, wherein the preset delay is determined based on the single-frame decoding delay when the decoder decodes multiple consecutive first images.

2. The image processing method as described in claim 1, characterized in that, Also includes: The rendering duration is determined based on the preset input frame rate; and When the single-frame decoding delay is greater than the rendering duration, it is determined that the terminal has the decoded cached image.

3. The image processing method as described in claim 1, characterized in that, The step of interpolating multiple consecutive frames of the first image before decoding on the terminal to generate multiple consecutive frames of the second image includes: Before decoding, the terminal receives a bitstream sent by a cloud server. The bitstream includes multiple frames of the first image and encoding information, including encoding method and resolution. Based on the encoding method and the resolution, N frames of the intermediate images are generated; N intermediate images are inserted between any two adjacent frames of the first image to generate multiple consecutive frames of the second image.

4. The image processing method as described in claim 3, characterized in that, The encoded information also includes a file number, an entropy encoding flag, an image width, and an image height; the frame interpolation operation performed on multiple consecutive frames of the first image before decoding at the terminal further includes: The encoding method is determined based on the gear position number and the entropy encoding flag; and The resolution is determined based on the image width and the image height.

5. The image processing method as described in claim 3, characterized in that, The encoding information also includes the maximum number of reference frames and the maximum number of decoded buffer images. When decoded buffer images exist on the terminal, the step of interpolating multiple consecutive first images to generate multiple consecutive second images includes: When the terminal has a decoded cache image, the maximum number of reference frames is less than a preset number, and the maximum decoded cache image is less than a preset decoded cache image, the frame interpolation operation is performed on multiple consecutive frames of the first image before the terminal performs decoding, so as to generate multiple consecutive frames of the second image.

6. The image processing method as described in claim 1, characterized in that, When a decoded cached image exists on the terminal, performing frame interpolation on multiple consecutive frames of the first image to generate multiple consecutive frames of the second image further includes: When the terminal has a decoded cached image, and the single-frame decoding delay is greater than the preset delay or the difference between the decoded output frame rate and the preset input frame rate is greater than the preset difference, the frame interpolation operation is performed on the cloud server connected to the terminal to generate multiple consecutive frames of the second image.

7. The image processing method as described in claim 6, characterized in that, The step of performing the frame interpolation operation on a cloud server communicatively connected to the terminal to generate multiple consecutive frames of the second image includes: Encode multiple consecutive frames of the first image to obtain multiple consecutive frames of the third image after encoding; The first image in each frame is encoded N times to generate N intermediate images corresponding to the first image in each frame, so that the third image, the first image and the N intermediate images correspond to each other. N target intermediate images are inserted between any two adjacent frames of the third image to generate multiple consecutive frames of the second image. The N target intermediate images are the N intermediate images corresponding to the previous frame in any two adjacent frames of the third image.

8. The image processing method as described in claim 1, characterized in that, Also includes: The interpolation input frame rate is determined based on the preset input frame rate and the value of N, wherein the preset input frame rate is the input frame rate when the decoder decodes multiple consecutive frames of the first image. The rendering duration is determined based on the interpolated input frame rate. The maximum number of interpolated frames is determined based on the rendering duration and the preset delay, wherein the preset delay is determined based on the single-frame decoding delay when the decoder decodes multiple consecutive frames of the first image. The value of N is determined based on the decoded cache image, including: The value of N is less than the maximum number of interpolated frames; or The value of N is equal to the number of decoded cache images.

9. The image processing method as described in claim 8, characterized in that, Decoding the second image includes: One frame of the second image is input to the decoder at intervals of the stated rendering duration; and After the decoder finishes decoding one frame of the second image, it decodes the next frame of the second image.

10. The image processing method according to any one of claims 1-9, characterized in that, Also includes: The cloud gaming client sends a request to the terminal to obtain the terminal's encoding and decoding capability information; According to the request, upload the encoding / decoding capability information; Based on the encoding / decoding capability information, the type of game running on the terminal, and network communication information, determine the target encoding / decoding configuration and target encoding / decoding strategy; The target encoding / decoding configuration and target encoding / decoding strategy are sent to the terminal through the cloud gaming client. Receive the target codec configuration and target codec strategy; Decoding the second image includes: The second image is decoded according to the target encoding / decoding configuration and target encoding / decoding strategy.

11. An image processing method applied to a terminal, characterized in that, include: When a decoding cache image exists on the terminal, a frame interpolation operation is performed on multiple consecutive frames of the first image before the terminal performs decoding to generate multiple consecutive frames of the second image. The frame interpolation operation includes inserting N intermediate images between any two adjacent frames of the first image. The intermediate images are generated based on the first images. The decoding cache image is an image cached in the decoder of the terminal. The value of N is determined based on the decoding cache image. The second image is decoded, and the single-frame decoding delay and decoding output frame rate of the decoder are obtained. The step of performing frame interpolation on multiple consecutive first images before decoding on the terminal to generate multiple consecutive second images when the terminal has a decoding cache image, the single-frame decoding delay is less than a preset delay, and the difference between the decoding output frame rate and the preset input frame rate is less than a preset difference, includes performing frame interpolation on multiple consecutive first images before decoding on the terminal to generate multiple consecutive second images, wherein the preset delay is determined based on the single-frame decoding delay when the decoder decodes multiple consecutive first images.

12. The image processing method as described in claim 11, characterized in that, The step of interpolating multiple consecutive frames of the first image to generate multiple consecutive frames of the second image includes: Receive a bitstream sent by a cloud server, the bitstream including multiple frames of the first image and encoding information, the encoding information including encoding method and resolution; Based on the encoding method and the resolution, N frames of the intermediate images are generated; N intermediate images are inserted between any two adjacent frames of the first image to generate multiple consecutive frames of the second image.

13. The image processing method as described in claim 12, characterized in that, The encoding information also includes the maximum number of reference frames and the maximum number of decoded buffer images. When decoded buffer images exist on the terminal, the step of interpolating multiple consecutive first images before decoding on the terminal to generate multiple consecutive second images includes: When the terminal has a decoded cache image, the maximum number of reference frames is less than a preset number, and the maximum decoded cache image is less than a preset decoded cache image, the frame interpolation operation is performed on multiple consecutive frames of the first image before the terminal performs decoding, so as to generate multiple consecutive frames of the second image.

14. The image processing method as described in claim 11, characterized in that, Also includes: Obtain the encoding and decoding capability information of the terminal; The client uploads the encoding / decoding capability information to the cloud server. The target codec configuration and target codec strategy are obtained from the cloud server. The target codec configuration and target codec strategy are determined by the cloud server based on the codec capability information, the type of game running on the terminal, and network communication information. Obtain multiple consecutive frames of the first image generated by the cloud server after encoding the original image according to the target encoding / decoding configuration and target encoding / decoding strategy; Decoding the second image includes: The second image is decoded according to the target encoding / decoding configuration and target encoding / decoding strategy.

15. An image processing method applied to a cloud server, characterized in that, include: When a decoded cache image is detected on a terminal connected to the cloud server, a frame interpolation operation is performed on multiple consecutive frames of the first image during encoding on the cloud server to generate multiple consecutive frames of the second image. The frame interpolation operation includes inserting N intermediate images between any two adjacent frames of the first image. The intermediate images are generated based on the first images. The decoded cache image is an image cached in the decoder of the terminal. The value of N is determined based on the decoded cache image. The encoded consecutive frames of the second image are sent to the terminal so that the terminal can decode the second image; The step of performing frame interpolation on multiple consecutive frames of the first image during encoding on the cloud server to generate multiple consecutive frames of the second image includes: encoding multiple consecutive frames of the first image to obtain encoded multiple consecutive frames of the third image. The first image is encoded N times in each frame to generate N intermediate images corresponding to each frame of the first image, so that the third image, the first image and the N intermediate images correspond to each other; N target intermediate images are inserted between any two adjacent frames of the third image to generate multiple consecutive frames of the second image, and the N target intermediate images are the N intermediate images corresponding to any frame in any two adjacent frames of the third image.

16. The image processing method as described in claim 15, characterized in that, Also includes: After the decoder finishes decoding the second image, the single-frame decoding delay of the decoder is obtained; and If the single-frame decoding delay is greater than the preset delay, the frame interpolation operation is stopped. The preset delay is determined based on the single-frame decoding delay when the decoder decodes multiple consecutive frames of the first image.

17. An image processing apparatus, characterized in that, The device includes: The first frame interpolation module is used to perform frame interpolation operations on multiple consecutive first images when there are decoded cache images in the terminal, so as to generate multiple consecutive second images. The frame interpolation operation includes inserting N intermediate images between any two adjacent first images. The intermediate images are generated based on the first images. The decoded cache images are images cached in the decoder of the terminal. The value of N is determined based on the decoded cache images. The first decoding module is used to decode the second image and obtain the single-frame decoding delay and decoding output frame rate during decoding. The first frame interpolation module is further configured to: when the terminal has a decoded cached image, the single-frame decoding delay is less than a preset delay, and the difference between the decoded output frame rate and the preset input frame rate is less than a preset difference, perform a frame interpolation operation on multiple consecutive first images before the terminal performs decoding to generate multiple consecutive second images, wherein the preset delay is determined based on the single-frame decoding delay when the decoder decodes multiple consecutive first images.

18. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executing the steps of the image processing method according to any one of claims 1-16 by calling the computer program stored in the memory.

19. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executing the image processing method as described in any one of claims 1-16 by calling the computer program stored in the memory.

20. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the image processing method according to any one of claims 1-16.

Images