Image processing method, device, system and computer readable storage medium

Abstract

The present disclosure provides an image processing method, device, system and computer-readable storage medium, which relate to the field of image processing technology. The method comprises: receiving rendering information and image identification of N frames of images sent by a VR terminal, the rendering information comprising first rendering information, the first rendering information comprising M first resolutions corresponding to M regions of each frame of image, the i-th region is adjacent to the i+1-th region, the first resolution corresponding to the i-th region is greater than the first resolution corresponding to the i+1-th region, 1≤i≤M‑1, M≥3, N≥1; acquiring the frame of image according to the image identification of each frame of image; rendering the N frames of image according to the first rendering information to obtain the N frames of rendered images comprises: rendering the M regions of each frame of image so that the M regions of each frame of image have the M first resolutions corresponding to each other; transmitting the N frames of rendered images to the VR terminal.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, system and computer-readable storage medium. Background Art

[0002] Virtual Reality (VR) technology can generate a highly realistic VR video, allowing users to immerse themselves in the virtual environment presented by the VR video. Through full immersion, users can look around the virtual environment and interact with objects in the virtual environment by moving.

[0003] In order to provide users with a 360-degree immersive experience, VR videos usually have higher resolutions and larger data volumes, resulting in higher requirements for network bandwidth during their transmission.

[0004] In the related art, the field of view (FOV) transmission scheme is usually used to transmit VR videos. In the FOV transmission scheme, for each frame of the VR video, only the user's main viewing area can be rendered with a relatively high resolution, while other areas are rendered with a relatively low resolution, thereby reducing the network bandwidth occupied by the higher resolution area and reducing the network bandwidth requirements for the transmission of VR videos. Summary of the invention

[0005] In the related art, the user experience is poor.

[0006] After analysis, it is found that in the relevant technologies, since the resolution of other areas except the user's main viewing area in each frame of image is extremely low, when the user's line of sight suddenly shifts from the main viewing area to these other areas, the user's visual perception of the surrounding environment within the real viewing angle may be reduced due to unclear vision, and may even cause dizziness and other physiological discomfort due to the inconsistency between visual perception and physical perception, resulting in a poor user experience.

[0007] In order to solve the above problems, the embodiments of the present disclosure propose the following solutions.

[0008] According to one aspect of an embodiment of the present disclosure, there is provided an image processing method, including: receiving rendering information and image identification of N frames of images sent by a virtual reality VR terminal, the rendering information including first rendering information, the first rendering information including M first resolutions corresponding to M regions of each frame of image, the i-th region is adjacent to the i+1-th region, the first resolution corresponding to the i-th region is greater than the first resolution corresponding to the i+1-th region, 1≤i≤M-1, M≥3, N≥1; acquiring the frame of image according to the image identification of each frame of image; rendering the N frames of image according to the first rendering information to obtain N frames of rendered images, including: rendering the M regions of each frame of image so that the M regions of each frame of image have M first resolutions corresponding to each other; transmitting the N frames of rendered images to the VR terminal.

[0009] In some embodiments, the j+1th region surrounds the jth region, where j is a value from 1 to M-1.

[0010] In some embodiments, the (i+1)th region surrounds the (i)th region.

[0011] In some embodiments, the first rendering information also includes size information and position information of M regions of each frame image, and rendering the M regions of each frame image includes: determining the M regions of each frame image according to the size information and position information of the M regions of the frame image; rendering the frame image according to M first resolutions that correspond one-to-one to the M regions of each frame image, so that the M regions of the frame image have M first resolutions that correspond one-to-one.

[0012] In some embodiments, the N frames of images include Q groups of images, each group of images includes an adjacent previous frame of image and a next frame of image, 1≤Q≤N-1, N≥2; the rendering information also includes second rendering information, the second rendering information includes a second resolution corresponding to a transition area between a first area of ​​the previous frame of image and a second area of ​​the next frame of image, the first area includes other areas in the previous frame of image except the Mth area, the second area includes other areas in the next frame of image except the Mth area, and the second resolution is between an average resolution of the first area and an average resolution of the second area; the method also includes: color rendering the transition area according to the second rendering information to obtain transition content with the second resolution; transmitting the transition content to the VR terminal so that the VR terminal displays the transition content between the previous frame of image and the next frame of image.

[0013] In some embodiments, the second resolution is between a minimum value of the first resolutions corresponding to the first area and a minimum value of the first resolutions corresponding to the second area.

[0014] In some embodiments, N≥2, transmitting the N frames of rendered images to the VR terminal includes: compressing the N frames of rendered images to obtain at least one compressed image, and transmitting each compressed image to the VR terminal, wherein each compressed image is obtained by compressing at least two consecutive frames of rendered images, and the at least two frames of rendered images in each compressed image are decompressed by the VR terminal and displayed frame by frame.

[0015] According to another aspect of an embodiment of the present disclosure, another image processing method is provided, including: receiving rendering information and image identification of N frames of images sent by a virtual reality VR terminal, the rendering information including two adjacent areas in each frame of image, two different resolutions corresponding to each other, N≥2; acquiring the frame of image according to the image identification of each frame of image; rendering the N frames of image according to the rendering information to obtain N frames of rendered images, including: rendering the two areas of each frame of image so that the two areas of each frame of image have the two resolutions corresponding to each other; compressing the N frames of rendered images to obtain at least one compressed image, and transmitting each compressed image to the VR terminal, wherein each compressed image is obtained by compressing at least two consecutive frames of rendered images, and the at least two frames of rendered images in each compressed image are decompressed by the VR terminal and displayed frame by frame.

[0016] In some embodiments, any compressed image is obtained in the following manner: encoding M regions in each rendered image of the at least two frames of rendered images in the any compressed image based on M different encoding levels, the encoding level corresponding to the i-th region is higher than the encoding level corresponding to the i+1-th region; compressing the encoded at least two frames of rendered images to obtain the any compressed image.

[0017] In some embodiments, the at least one compressed image includes a first compressed image and a second compressed image, the N-frame rendered image includes a continuous multi-frame rendered image, and the multi-frame rendered image is composed of at least two continuous frames of first rendered images and at least two continuous frames of second rendered images; compressing the N-frame rendered image to obtain at least one compressed image, and transmitting each compressed image to the VR terminal includes: compressing the at least two frames of first rendered images to obtain the first compressed image; after obtaining the first compressed image, transmitting the first compressed image to the VR terminal, and compressing the at least two frames of second rendered images to obtain the second compressed image, wherein the sum of the time of transmitting the first compressed image and the time of displaying the first compressed image by the VR terminal is greater than or equal to the sum of the time of obtaining the second compressed image and the time of transmitting the second compressed image.

[0018] In some embodiments, each compressed image has the same capacity as one frame of rendered image.

[0019] In some embodiments, each compressed image is obtained by compressing three consecutive frames of rendered images.

[0020] In some embodiments, the N frames of images are rendered in parallel.

[0021] In some embodiments, the M regions of each frame image are rendered in parallel.

[0022] In some embodiments, each compressed image carries a quality of service QoS label, and transmitting each compressed image to the VR terminal includes: transmitting each compressed image to the VR terminal using a network having a transmission quality corresponding to the QoS label.

[0023] According to another aspect of an embodiment of the present disclosure, another image processing method is provided, including: obtaining historical viewing information of each frame of N frames of images; determining M regions in the frame of image based on the historical viewing information of each frame of image, the i-th region is adjacent to the i+1-th region, 1≤i≤M-1, M≥3, N≥1; determining first rendering information, the first rendering information including M first resolutions corresponding one-to-one to the M regions of each frame of image, the first resolution corresponding to the i-th region being greater than the first resolution corresponding to the i+1-th region; sending the image identification of the N frames of image and the rendering information including the first rendering information to a cloud server, so that the cloud server renders the N frames of image according to the rendering information and the image identification of the N frames of image.

[0024] In some embodiments, the first rendering information further includes size information and position information of M regions of each frame image.

[0025] In some embodiments, where N≥2, determining the M regions in each frame image based on the historical viewing information of the frame image includes: determining the M regions of the first frame image and the M initial regions of each frame image in the N-1 frames of the N frames of image other than the first frame image based on the historical viewing information of the frame image; obtaining historical eye tracking data; adjusting the positions of the other initial regions in the k+1 frame image other than the M initial region based on the centers of the other regions in the k-th frame image other than the M-th region and the historical eye tracking data, 1≤k≤N-1; and determining the M-th initial region in the k+1-th frame image and the adjusted other initial regions as the M regions of the k+1-th frame image.

[0026] In some embodiments, adjusting the positions of other initial regions except the Mth initial region in the k+1th frame image includes: predicting the viewpoint of the k+1th frame image based on the centers of other regions except the Mth region in the kth frame image and the historical eye tracking data; adjusting the positions of the other initial regions so that the centers of the adjusted other initial regions coincide with the viewpoint.

[0027] In some embodiments, the N frames of images include Q groups of images, each group of images includes an adjacent previous frame image and a next frame image, the rendering information also includes a second resolution corresponding to a transition area between a first area of ​​the previous frame image and a second area of ​​the next frame image, the first area includes other areas in the previous frame image except the Mth area, the second area includes other areas in the next frame image except the Mth area, 1≤Q≤N-1, N≥2, the method also includes: determining a second resolution corresponding to the transition area, the second resolution is between an average resolution of the first area and an average resolution of the second area; wherein the rendering information also includes the second resolution.

[0028] According to another aspect of the embodiments of the present disclosure, there is provided an image processing device, including: a receiving module, configured to receive rendering information and image identification of N frames of images sent by a virtual reality VR terminal, the rendering information including first rendering information, the first rendering information including M first resolutions corresponding to M regions of each frame of image, the i-th region is adjacent to the i+1-th region, the first resolution corresponding to the i-th region is greater than the first resolution corresponding to the i+1-th region, 1≤i≤M-1, M≥3, N≥1; an acquisition module, configured to acquire the frame of image according to the image identification of each frame of image; a rendering module, configured to render the N frames of image according to the first rendering information to obtain N frames of rendered images, including: rendering the M regions of each frame of image so that the M regions of each frame of image have M first resolutions corresponding to each other; a transmission module, configured to transmit the N frames of rendered images to the VR terminal.

[0029] According to another aspect of the embodiments of the present disclosure, there is provided an image processing device, comprising: a receiving module, configured to receive rendering information and image identification of N frames of images sent by a virtual reality VR terminal, the rendering information comprising two adjacent regions in each frame of image, two different resolutions corresponding to each other, N≥2; an acquisition module, configured to acquire each frame of image according to the image identification of the frame of image; a rendering module, configured to render the N frames of image according to the rendering information to obtain N frames of rendered images, comprising: rendering the two regions of each frame of image so that the two regions of each frame of image have the two resolutions corresponding to each other; a transmission module, configured to compress the N frames of rendered images to obtain at least one compressed image, and transmit each compressed image to the VR terminal, wherein each compressed image is obtained by compressing at least two consecutive frames of rendered images, and the at least two frames of rendered images in each compressed image are decompressed by the VR terminal and displayed frame by frame.

[0030] According to another aspect of an embodiment of the present disclosure, there is provided an image processing device, comprising: a memory; and a processor coupled to the memory, configured to execute the method described in any one of the above embodiments based on instructions stored in the memory.

[0031] According to another aspect of an embodiment of the present disclosure, there is provided an image processing device, comprising: an acquisition module, configured to acquire historical viewing information of each frame of N frames of images; a first determination module, configured to determine M regions in the frame of image based on the historical viewing information of each frame of image, the i-th region is adjacent to the i+1-th region, 1≤i≤M-1, M≥3, N≥1; a second determination module, configured to determine first rendering information, the first rendering information comprising M first resolutions corresponding one-to-one to the M regions of each frame of image, the first resolution corresponding to the i-th region being greater than the first resolution corresponding to the i+1-th region; a sending module, configured to send the image identification of the N frames of image and the rendering information including the first rendering information to a cloud server, so that the cloud server renders the N frames of image according to the rendering information and the image identification of the N frames of image.

[0032] According to another aspect of the embodiments of the present disclosure, a cloud server is provided, comprising the image processing device described in any one of the above embodiments.

[0033] According to another aspect of the embodiments of the present disclosure, a VR terminal is provided, comprising the image processing device described in any one of the above embodiments.

[0034] According to another aspect of the embodiments of the present disclosure, there is provided an image processing system, comprising: the cloud server described in any one of the above embodiments; and the VR terminal described in any one of the above embodiments.

[0035] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, comprising computer program instructions, wherein when the computer program instructions are executed by a processor, the method described in any one of the above embodiments is implemented.

[0036] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0037] In the disclosed embodiment, by rendering the M regions of each frame of N frames, the M regions of each frame of image can have one-to-one corresponding M first resolutions. In this way, since different regions in each frame of image have different resolutions, there is at least one intermediate resolution region between the highest resolution region and the lowest resolution region of each frame of image, which can allow the user to adapt to the resolution change in the process of the line of sight shifting from the highest resolution region to the lowest resolution region, thereby improving the user's visual perception of the surrounding environment within the real viewing angle, thereby improving the user experience.

[0038] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 is a flowchart of an image processing method according to some embodiments of the present disclosure;

[0041] Figure 2 is a schematic diagram of a transition region according to some embodiments of the present disclosure;

[0042] Figure 3 is a flowchart of an image processing method according to some other embodiments of the present disclosure;

[0043] Figure 4 is a flowchart of an image processing method according to some other embodiments of the present disclosure;

[0044] Figure 5 is a flowchart of an image processing method according to some further embodiments of the present disclosure;

[0045] Figure 6 is a flowchart of some implementations of step 504 according to some embodiments of the present disclosure;

[0046] Figure 7 is a schematic structural diagram of an image processing device according to some embodiments of the present disclosure;

[0047] Figure 8 is a schematic structural diagram of an image processing device according to some other embodiments of the present disclosure;

[0048] Fig. 9 is a schematic structural diagram of an image processing device according to some other embodiments of the present disclosure;

[0049] Fig.10 is a schematic structural diagram of an image processing device according to some further embodiments of the present disclosure;

[0050] Fig.11 is a schematic diagram of the structure of an image processing system according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0052] Unless specifically stated otherwise, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0053] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0054] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0055] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0056] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0057] Figure 1It is a flowchart of an image processing method according to some embodiments of the present disclosure.

[0058] In step 102, rendering information and image identifiers of N frames of images sent by the VR terminal are received.

[0059] Here, the rendering information may include first rendering information. The first rendering information may include M first resolutions corresponding to M regions of each frame of image. The i-th region may be adjacent to the i+1-th region, the first resolution corresponding to the i-th region may be greater than the first resolution corresponding to the i+1-th region, 1≤i≤M-1, M≥3, N≥1. In some implementations, the rendering information may also include other information, which will be described later.

[0060] In some embodiments, different regions may correspond to different user interests. For example, the user interest corresponding to the i-th region of each frame image may be greater than the user interest corresponding to the i+1-th region. For example, a frame image is divided into three regions with user interests from high to low, the resolution of the first region may be 12k, the resolution of the second region may be 8k, and the resolution of the third region may be 4k.

[0061] In step 104, each frame of image is acquired according to the image identifier of the frame of image.

[0062] For example, each frame of image may be downloaded from cloud storage according to the image identifier of the frame of image.

[0063] In step 106 , N frames of images are rendered according to the first rendering information to obtain N frames of rendered images.

[0064] Here, in step 106, the M regions of each frame of image are rendered so that the M regions of each frame of image have one-to-one corresponding M first resolutions, thereby obtaining N frames of rendered images.

[0065] In some embodiments, the VR terminal may determine the M regions of each frame image, and the first rendering information sent by the VR terminal may further include size information and position information of the M regions of each frame image. The manner in which the VR terminal determines the M regions of each frame image will be described below.

[0066] In this case, at this time, the M regions of each frame image can be determined according to the size information and position information of the M regions of the frame image, and then the frame image is rendered according to the M first resolutions corresponding to the M regions of each frame image, so that the M regions of the frame image have the M first resolutions corresponding to each other. For example, a frame image can include 3 regions, and after rendering the frame image, the resolutions of the 3 regions can be 12k, 8k and 4k respectively.

[0067] In step 108, N frames of rendered images are transmitted to the VR terminal.

[0068] In some embodiments, the N frames of rendered images may be transmitted frame by frame to the VR terminal so that the VR terminal displays the N frames of images frame by frame. In other embodiments, the N frames of rendered images may be transmitted to the VR terminal in other ways, which will be described in detail below.

[0069] In the above embodiment, by rendering the M regions of each frame of N frames, the M regions of each frame of image can have one-to-one corresponding M first resolutions. In this way, since different regions in each frame of image have different resolutions, there is at least one intermediate resolution region between the highest resolution region and the lowest resolution region of each frame of image, which can allow the user to adapt to the resolution change in the process of the line of sight shifting from the highest resolution region to the lowest resolution region, thereby improving the user's visual perception of the surrounding environment within the real viewing angle, thereby improving the user experience.

[0070] In some embodiments, the j+1th region may surround the jth region, where j is a value from 1 to M-1. For example, the first region is a circular region, and the second region may be a region excluding the circular region in a circumscribed triangle of the circular region, or the second region may be a circular ring region between a circle having the same center as the circular region but a radius greater than the circular region and the circular region.

[0071] In some other embodiments, the i+1th region surrounds the ith region. In this way, when the user's line of sight shifts from any direction of the higher resolution region to the lower resolution region, it can be ensured that the resolution change perceived by the user before and after the line of sight shifts is small, thereby further improving the user experience.

[0072] In some embodiments, N frames of images may include Q groups of images, each group of images may include an adjacent previous frame of image and a next frame of image, 1≤Q≤N-1, N≥2.

[0073] In this case, the rendering information sent by the VR terminal may also include second rendering information. The second rendering information may include a second resolution corresponding to a transition area between a first area of ​​a previous frame image and a second area of ​​a next frame image in two adjacent frames of images in each group of images.

[0074] Figure 2 is a schematic diagram of a transition region according to some embodiments of the present disclosure.

[0075] like Figure 2 As shown, the previous frame image and the next frame image both include M regions ( Figure 2Three regions are schematically shown, and are labeled as regions A, B, and C in descending order of resolution).

[0076] Region A in the previous frame image is a hexagon, region B is a region consisting of four triangles adjacent to the four sides of the hexagon, and region C is a region other than regions A and B. The first region 21 in the previous frame image may include other regions in the previous frame image 21 except the Mth region (i.e., the region consisting of region A and region B in the previous frame image 21).

[0077] The area A in the next frame image is a rhombus, the area B is an area consisting of four triangles adjacent to the four sides of the rhombus, and the area C is an area other than the areas A and B. The second area 22 in the next frame image may include other areas in the next frame image except the Mth area (i.e., an area consisting of the area A and the area B in the next frame image).

[0078] like Figure 2 As shown in the middle shaded portion, the transition area 23 may be located between the first area 21 of the previous frame image and the second area 22 of the next frame image.

[0079] It should be understood that the Mth region in the previous frame image and the next frame image are both regions with the lowest resolution. That is, the first region is the region in the previous frame image except the region with the lowest resolution, and the second region is the region in the next frame image except the region with the lowest resolution.

[0080] In some cases, if bandwidth resources are sufficient, the second resolution may be between an average resolution of the first region and an average resolution of the second region.

[0081] In other cases, if bandwidth resources are limited, the second resolution may be between the minimum value of the first resolution corresponding to the first area and the minimum value of the first resolution corresponding to the second area. In this way, even if bandwidth resources are limited, the second resolution may still be determined to achieve rendering of the transition area.

[0082] Figure 3 It is a flowchart of an image processing method according to some other embodiments of the present disclosure.

[0083] and Figure 1 Compared with the embodiment shown, Figure 3 The illustrated embodiment further includes steps 302 to 304 .

[0084] In step 302, color rendering is performed on the transition area according to the second rendering information to obtain transition content with a second resolution.

[0085] In some embodiments, the transition content may include a transition color determined based on the color of the previous frame image and the color of the next frame image. For example, an intermediate value between the average chromaticity of the previous frame image and the average chromaticity of the next frame image may be determined, and the chromaticity of the intermediate value may be determined as the transition color in the transition content.

[0086] In step 304, the transition content is transmitted to the VR terminal so that the VR terminal displays the transition content between the previous frame image and the next frame image.

[0087] In some embodiments, the transition content and N frames of rendered images may be transmitted to the VR terminal together.

[0088] In the above embodiment, when N frames of images are multiple frames of images, considering that in the process of switching between the previous and next frames of images, the user's line of sight is usually switched from the area other than the lowest resolution area in the previous frame of image to the area other than the lowest resolution area in the next frame of image, by increasing the color rendering of the transition area between the two areas, the VR terminal can display the transition content corresponding to the transition area between the previous frame of image and the next frame of image. In this way, the VR terminal can further display the transition content with transition colors for the user between the display of the previous and next frames, thereby making the switching of the screen smoother, further improving the visual perception of the user within the real viewing angle, and thus further improving the user experience.

[0089] In some embodiments, N frames of images may be rendered in parallel. In other embodiments, N frames of images and the transition area of ​​each group of images may be rendered in parallel. In this way, the rendering efficiency of the image may be improved.

[0090] In some embodiments, the M regions of each frame of image may be rendered in parallel, thereby further improving the image rendering efficiency.

[0091] In some other embodiments, on the basis of rendering N frames of images in parallel, M regions of each frame of images may be rendered in parallel, so that the image rendering efficiency may be further improved.

[0092] In the case where the N frames of images are multiple frames, the rendered N frames of images can be transmitted to the VR terminal frame by frame. However, due to inevitable network delays and limited transmission bandwidth, the VR terminal may not receive the next frame of image after displaying the previous frame of image, resulting in jittery images, which leads to poor user experience.

[0093] To solve this problem, in the related art, a VR terminal can generate an intermediate image (hereinafter referred to as the "intermediate frame") between displaying the previous frame image and the next frame image, and display the intermediate frame when the next frame image is not received to avoid image jitter.

[0094] However, the user experience in related technologies is still poor.

[0095] After analysis, it was found that the intermediate frame generated by the real-time rendering of the VR terminal is obtained by vectorizing and re-rendering the previous frame based on the predicted posture transformation of the user's head. In other words, the new content in the next frame does not exist in this intermediate frame. In this case, the area where the new content should be presented to the user can only be repeatedly filled with the content in the previous frame, which leads to another derivative problem, that is, the intermediate frame will have a distorted ghosting phenomenon, resulting in a poor user experience.

[0096] Based on this, the present disclosure also provides the following solution.

[0097] When N≥2, N frames of rendered images can also be transmitted to the VR terminal in the following manner.

[0098] The N frames of rendered images may be compressed to obtain at least one compressed image, and each compressed image may be transmitted to the VR terminal. Each compressed image may be obtained by compressing at least two consecutive frames of rendered images, and at least two frames of rendered images in each compressed image may be decompressed by the VR terminal and displayed frame by frame.

[0099] It should be understood that, for any two compressed images, one of the compressed images is obtained by compressing certain frame rendering images, and the other compressed image is obtained by compressing other frame rendering images that are completely different from the certain frame rendering images.

[0100] In this way, compared with the frame-by-frame transmission method, not only can the above-mentioned derivative problems be solved, but also the total bandwidth resources required for transmitting N frames of images can be saved.

[0101] Next, we will combine Figure 4 The illustrated embodiment is further described.

[0102] Figure 4 is a flowchart of an image processing method according to some other embodiments of the present disclosure.

[0103] In step 402, rendering information and image identifiers of N frames of images sent by the VR terminal are received.

[0104] Here, the rendering information may include two mutually different resolutions corresponding to two adjacent regions in each frame of image, where N≥2.

[0105] In step 404, each frame of image is acquired according to the image identifier of the frame of image.

[0106] In step 406, N frames of images are rendered according to the rendering information to obtain N frames of rendered images. Here, in step 406, two regions of each frame of image are rendered so that the two regions of each frame of image have two resolutions corresponding to each other, thereby obtaining N frames of rendered images.

[0107] The specific implementation of steps 402 to 406 can refer to the related description of the aforementioned steps 102 to 106, which will not be repeated here.

[0108] In step 408, the N frames of rendered images are compressed to obtain at least one compressed image, and each compressed image is transmitted to the VR terminal.

[0109] Here, each compressed image may be obtained by compressing at least two consecutive frames of rendered images, and at least two frames of rendered images in each compressed image may be decompressed by the VR terminal and displayed frame by frame.

[0110] In some embodiments, each compressed image can be obtained by compressing three consecutive frames of rendered images. In this way, when the VR terminal displays one frame of the compressed image, there are still two frames of images cached locally for direct call, thereby greatly improving the tolerance to latency while taking into account limited bandwidth resources.

[0111] Assume that each compressed image can be obtained by compressing three consecutive frames of rendered images. For example, compressing six frames of rendered images can obtain two compressed images, and the two compressed images are transmitted to the VR terminal. For another example, compressing four frames of rendered images can only obtain one compressed image. At this time, this compressed image and the remaining uncompressed rendered image can be transmitted to the VR terminal in sequence, or only this compressed image can be transmitted to the VR terminal. After obtaining two other frames of rendered images consecutive to this uncompressed rendered image next time, these three frames of rendered images are compressed and then sent to the VR terminal.

[0112] In other words, a single frame of rendered image may be transmitted between two adjacent compressed images. In some cases, continuous compressed images may also be transmitted, which will be described later.

[0113] In the above embodiment, each time a compressed image is transmitted to the VR terminal, the VR terminal can receive at least two consecutive frames of rendered images, so that when the VR terminal displays one frame of the rendered image, at least one frame of the rendered image is still cached locally. In this way, even if the VR terminal does not have time to receive the next frame of the image due to network delay, the VR terminal can directly call the cached rendered image from the local for display without generating intermediate frames, which not only avoids the jitter of the picture, but also reduces the resource consumption of the VR terminal for generating intermediate frames, and there will be no distorted ghosting phenomenon, thereby improving the user experience.

[0114] In some embodiments, any compressed image is obtained as follows.

[0115] First, M regions in each rendered image of at least two rendered images in any one compressed image may be encoded based on M different encoding levels.

[0116] Here, the coding level corresponding to the i-th region may be higher than the coding level corresponding to the i+1-th region.

[0117] For example, each frame of the rendered image may be encoded using the H.265 encoding standard. The first region corresponds to the highest encoding level, and the Mth region corresponds to the lowest encoding level.

[0118] Then, the encoded at least two frames of rendered images are compressed to obtain any compressed image.

[0119] In this way, for each of at least two rendered images in any compressed image, the M areas in the frame image can be hierarchically encoded using M different encoding levels, thereby reducing the resource consumption caused by uniformly using a higher encoding level for encoding while ensuring the reliability of the encoding.

[0120] In some embodiments, the N frames of rendered images may include a plurality of consecutive frames of rendered images, and the plurality of consecutive frames of rendered images may be composed of at least two consecutive frames of first rendered images and at least two consecutive frames of second rendered images.

[0121] In this case, taking the example that at least one compressed image obtained by compressing N frames of rendered images includes a first compressed image and a second compressed image, step 408 is further described as follows.

[0122] First, at least two frames of first rendered images may be compressed to obtain first compressed images.

[0123] Then, after obtaining the first compressed image, the first compressed image is transmitted to the terminal, and at least two frames of the second rendered image are compressed to obtain a second compressed image.

[0124] Here, the sum of the time for transmitting the first compressed image and the time for the VR terminal to display the first compressed image may be greater than or equal to the sum of the time for obtaining the second compressed image and the time for transmitting the second compressed image.

[0125] In this way, it can be ensured that the VR terminal can receive the second compressed image when displaying the last frame of the rendered image of the first compressed image, and then can directly decompress the second compressed image and continue to display at least two frames of the rendered image frame by frame, thereby ensuring that the VR terminal can still display rendered images containing new content after displaying all the rendered images in the first compressed image, further reducing the resource consumption of the VR terminal for generating intermediate frames and further improving the user experience.

[0126] In some embodiments, the total number of frames of rendered images contained in at least one compressed image obtained by compressing N frames of rendered images may be equal to N. That is, after compressing each frame of the N frames of rendered images, all the images obtained may be compressed images without leaving a separate rendered image, so that the compressed images can be continuously transmitted to the VR terminal. In this way, it can be ensured that the VR terminal continuously displays multiple compressed images, further reducing the resource consumption of the VR terminal for generating intermediate frames, and further improving the user experience.

[0127] In some embodiments, each compressed image has the same capacity as a frame of rendered image. In this way, it can be ensured that the compressed image has the same transmission rate and bandwidth occupancy as a frame of rendered image, and multiple frames of images can be transmitted to the VR terminal without additional resource consumption.

[0128] In some embodiments, each compressed image may carry a Quality of Service (QoS) tag. In this case, each compressed image may be transmitted to the VR terminal using a network having a transmission quality corresponding to the QoS tag. In this way, the network side may transmit the compressed image according to the transmission quality corresponding to the QoS tag carried by each compressed image during transmission, thereby improving the reliability of transmission.

[0129] The methods in the above-mentioned related embodiments can all be implemented by a cloud server. Next, the methods that can be executed by the VR terminal will be described.

[0130] Figure 5 It is a flowchart of an image processing method according to some further embodiments of the present disclosure.

[0131] In step 502, historical viewing information of each frame of the N frames of images is obtained.

[0132] In some embodiments, the historical viewing information of each frame of image may include historical user viewing information of the frame of image. The historical viewing information may reflect the historical user's interest points or areas of interest for each frame of image.

[0133] In step 504, M regions in each frame of image are determined based on the historical viewing information of the frame of image.

[0134] Here, the i-th region is adjacent to the i+1-th region, 1≤i≤M-1, M≥3, N≥1.

[0135] In some embodiments, based on the historical viewing information of each frame of image, multiple regions with different user interest levels in the frame of image may be determined, and different regions may correspond to different user interest levels. For example, the user interest level corresponding to the i-th region of each frame of image may be greater than the user interest level corresponding to the i+1-th region.

[0136] In some embodiments, feature points (e.g., points where the grayscale value of the image changes suddenly) in each frame of the image can be extracted through a machine learning model, and these feature points can be classified based on an unsupervised learning clustering algorithm (or a supervised learning K-nearest neighbor classification algorithm) to determine the M regions of the frame of the image. For example, a region can be a region covering a certain type of feature points.

[0137] It should be understood that the above steps 502 to 504 can also be completed by the cloud server alone.

[0138] In step 506, first rendering information is determined, where the first rendering information includes M first resolutions corresponding to the M regions of each frame of image.

[0139] Here, the first resolution corresponding to the i-th region is greater than the first resolution corresponding to the (i+1)-th region.

[0140] In some embodiments, the VR terminal may determine the M first resolutions corresponding to the M regions of each frame of image according to the actual allocable bandwidth resources. For example, when bandwidth resources are limited, three first resolutions corresponding to three regions of a frame of image may be determined based on the bandwidth resources according to the allocation ratio of 5:3:2 (for example, 8k, 4k, and 2k).

[0141] In step 508, the image identifier of the N frames of images and the rendering information including the first rendering information are sent to the cloud server so that the cloud server renders the N frames of images according to the rendering information and the image identifier of the N frames of images. In some implementations, the rendering information sent to the cloud server may also include other information, which will be described later.

[0142] In some embodiments, the first rendering information determined by the VR terminal may also include size information and position information of the M regions of each frame image, so that the cloud server can accurately determine the M regions in each frame image according to the first rendering information. For example, the size information includes at least one of shape information and size information.

[0143] In some embodiments, the cloud server can Figures 1 to 4 The rendering method in the related embodiment shown renders N frames of images.

[0144] It should be noted that the user interest determined based on historical viewing information is a statistical standard derived from big data. For most users, it can usually be considered that their main viewing area is the area with the highest user interest. However, in some cases, considering individual differences, for some users, their main viewing area may also be the area with the second highest or lower user interest, rather than the area with the highest user interest. However, in the related art, only the area with the highest user interest is rendered in high resolution, ignoring the experience of other users.

[0145] In view of this, in the above method provided by the present disclosure, by determining the M different resolutions corresponding to the M different regions of each frame image, and sending the rendering information containing this information to the cloud server, so that the cloud server can render each frame image with M different resolutions. In this way, in addition to the highest resolution region in each frame image, there are at least some regions with higher resolutions, so that the feelings of other users under individual differences can be taken into account, further improving the user experience.

[0146] When N≥2, step 504 can also be implemented as follows.

[0147] Figure 6 is a flowchart of some implementations of step 504 according to some embodiments of the present disclosure.

[0148] In step 602, M regions of a first frame image and M initial regions of each frame image in N-1 frames of images except the first frame image among N frames of images are determined based on historical viewing information of each frame image.

[0149] It should be understood that step 602 may be implemented in the manner shown in the related embodiment of step 504 .

[0150] At step 604, historical eye tracking data is obtained.

[0151] In some embodiments, the historical eye tracking data may be historical user eye tracking data obtained through eye tracking technology. The historical eye tracking data may reflect the deviation trajectory of the user's line of sight in the process of switching from a previous viewpoint to a subsequent viewpoint.

[0152] In step 606, based on the centers of the other regions except the Mth region in the kth frame image and the historical eye tracking data, the positions of the other initial regions except the Mth initial region in the k+1th frame image are adjusted.

[0153] Here, 1≤k≤N-1.

[0154] In some embodiments, step 606 may be implemented as follows.

[0155] First, the viewpoint of the k+1th frame image can be predicted based on the centers of other regions except the Mth region in the kth frame image and historical eye tracking data.

[0156] For example, based on the centers of areas other than the Mth area in the first frame and historical eye tracking data, the viewpoint of the second frame can be predicted; based on the centers of areas other than the Mth area in the second frame and historical eye tracking data, the viewpoint of the third frame can be predicted... and so on.

[0157] Then, the positions of other initial regions in the k+1th frame image may be adjusted so that the centers of the adjusted other initial regions in the k+1th frame image coincide with the predicted viewpoint.

[0158] It should be understood that the other regions except the Mth region in the kth frame image are the other regions except the lowest resolution region in the frame image.

[0159] In step 608, the Mth initial region in the k+1th frame image and the other adjusted initial regions are determined as the M regions of the k+1th frame image.

[0160] It should be understood that, for the first frame image, its M initial regions are M regions, and there is no need to adjust them in the manner of steps 604 to 608 .

[0161] In the above embodiment, for each frame of the N-1 frames, after determining the M initial areas of each frame, the M initial areas of each frame can be adjusted in combination with historical eye tracking data to make the determined M areas of each frame more accurate, thereby further improving the user experience.

[0162] In some embodiments, N frames of images may include Q groups of images, each group of images may include an adjacent previous frame of image and a next frame of image, 1≤Q≤N-1, N≥2.

[0163] In this case, the VR terminal can also determine the second resolution corresponding to the transition area between the first area of ​​the previous frame image and the second area of ​​the next frame image of each group of images according to the allocable bandwidth resources. For example, the second resolution can be between the average resolution of the first area and the average resolution of the second area.

[0164] Here, the first region may include other regions except the Mth region in the previous frame image, and the second region may include other regions except the Mth region in the next frame image. For a detailed description of the transition region, please refer to Figure 2 The related embodiments shown will not be described in detail here.

[0165] In some embodiments, the VR terminal may also send rendering information of the second resolution corresponding to the transition area in each group of images to the cloud server, so that the cloud server can Figure 3 The method in the embodiment shown is used to render the transition area. For details, please refer to Figure 3 The illustrated embodiments will not be described in detail here.

[0166] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0167] Figure 7 is a schematic structural diagram of an image processing device according to some embodiments of the present disclosure.

[0168] like Figure 7 As shown, the image processing device 700 includes a receiving module 701 , an acquiring module 702 , a rendering module 703 and a transmitting module 704 .

[0169] The receiving module 701 may be configured to receive rendering information and image identifiers of N frames of images sent by the VR terminal. The rendering information may include first rendering information, and the first rendering information may include M first resolutions corresponding to M regions of each frame of the image, the i-th region may be adjacent to the i+1-th region, the first resolution corresponding to the i-th region may be greater than the first resolution corresponding to the i+1-th region, 1≤i≤M-1, M≥3, N≥1.

[0170] The acquisition module 702 may be configured to acquire each frame of image according to the image identifier of the frame of image.

[0171] The rendering module 703 may be configured to render N frames of images according to the first rendering information to obtain N frames of rendered images. The rendering module 703 may be configured to render the M regions of each frame of images so that the M regions of each frame of images have one-to-one corresponding M first resolutions, thereby obtaining N frames of rendered images.

[0172] The transmission module 704 may be configured to transmit N frames of rendered images to the VR terminal.

[0173] Figure 8 is a schematic structural diagram of an image processing device according to some other embodiments of the present disclosure.

[0174] like Figure 8 As shown, the image processing device 800 includes a receiving module 801 , an acquiring module 802 , a rendering module 803 and a transmitting module 804 .

[0175] The receiving module 801 may be configured to receive rendering information and image identifiers of N frames of images sent by the VR terminal, wherein the rendering information includes two different resolutions corresponding to two adjacent regions in each frame of the image, where N≥2.

[0176] The acquisition module 802 may be configured to acquire each frame of image according to the image identifier of the frame of image.

[0177] The rendering module 803 may be configured to render N frames of images according to the rendering information to obtain N frames of rendered images, including: rendering two regions of each frame of image so that the two regions of each frame of image have two resolutions corresponding to each other;

[0178] The transmission module 804 may be configured to compress N frames of rendered images to obtain at least one compressed image, and transmit each compressed image to the VR terminal. Each compressed image is obtained by compressing at least two consecutive frames of rendered images, and at least two frames of rendered images in each compressed image are decompressed by the VR terminal and displayed frame by frame.

[0179] Fig. 9 is a schematic structural diagram of an image processing device according to some other embodiments of the present disclosure.

[0180] like Fig. 9 As shown, the image processing device 900 includes a memory 901 and a processor 902 coupled to the memory 901, and the processor 902 is configured to execute the method of any of the above embodiments (for example, Figures 1 to 4 The method in the embodiment shown, or, Figures 5 and 6 method in the illustrated embodiment).

[0181] The memory 901 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0182] The image processing device 900 may also include an input / output interface 903, a network interface 904, a storage interface 905, etc. These interfaces 903, 904, 905, and the memory 901 and the processor 902 may be connected, for example, via a bus 906. The input / output interface 903 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 904 provides a connection interface for various networked devices. The storage interface 905 provides a connection interface for external storage devices such as an SD card and a USB flash drive.

[0183] Fig.10 is a schematic structural diagram of an image processing device according to some further embodiments of the present disclosure.

[0184] like Fig.10 As shown, the image processing apparatus 1000 includes an acquisition module 1001 , a first determination module 1002 , a second determination module 1003 and a sending module 1004 .

[0185] The acquisition module 1001 may be configured to acquire historical viewing information of each frame of the N frames of images.

[0186] The first determination module 1002 may be configured to determine M regions in each frame of image based on historical viewing information of the frame of image, where the i-th region is adjacent to the i+1-th region, 1≤i≤M-1, M≥3, and N≥1.

[0187] The second determination module 1003 may be configured to determine first rendering information, wherein the first rendering information includes M first resolutions corresponding to M regions of each frame image, and the first resolution corresponding to the i-th region is greater than the first resolution corresponding to the i+1-th region.

[0188] The sending module 1004 may be configured to send the image identification of the N frames of images and the rendering information including the first rendering information to the cloud server, so that the cloud server renders the N frames of images according to the rendering information and the image identification of the N frames of images.

[0189] The disclosed embodiment further provides a cloud server, comprising the image processing device of any one of the above embodiments (eg, the image processing device 700 / 800 / 900).

[0190] The disclosed embodiment also provides a VR terminal, comprising an image processing device according to any one of the above embodiments (eg, image processing device 900 / 1000).

[0191] Fig.11 is a schematic diagram of the structure of an image processing system according to some embodiments of the present disclosure.

[0192] like Fig.11 As shown, the image processing system 1100 may include a cloud server 1101 and a VR terminal 1102 of any one of the above-mentioned embodiments.

[0193] In some embodiments, the VR terminal 1102 may include a wireless network connector, a sensor, a player, a decoder, a renderer, and a display screen.

[0194] The wireless network connector may be configured to access a wireless network in order to send or receive data information via the wireless network.

[0195] The sensor may be configured to capture the posture of the user's head. For example, the captured posture of the user's head may be used as auxiliary information to predict the viewpoint.

[0196] The Android application package (APK) player may be provided with the image processing device (e.g., image processing device 900 / 1000) of any of the above embodiments. In some embodiments, the APK player may also be provided with a data buffer to cache compressed images sent by the cloud server 1101.

[0197] The decoder may be configured to decode the received image. For example, the decoder may be configured to decompress a received compressed image to obtain at least two frames of rendered images. The decoder may also be configured to decode each frame of rendered images and transmit each decoded frame of images to a renderer for secondary rendering or to a display screen for display.

[0198] The renderer may be configured to perform secondary rendering on each frame of image based on the user posture captured by the sensor, so that each frame of image may be more adapted to the field of view under the current user posture.

[0199] The embodiments of the present disclosure further provide a computer-readable storage medium, comprising computer program instructions, which implement the method of any one of the above embodiments when executed by a processor.

[0200] The embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor, the method of any one of the above embodiments is implemented.

[0201] So far, various embodiments of the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.

[0202] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0203] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions specified in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0206] Although some specific embodiments of the present disclosure have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified or some technical features may be replaced by equivalents without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. An image processing method, comprising: Receive rendering information and image identifiers of N frames of images sent by a virtual reality VR terminal, wherein the N frames of images include Q groups of images, each group of images includes an adjacent previous frame of image and a next frame of image, 1≤Q≤N-1, N≥2, and the rendering information includes first rendering information and second rendering information, wherein, The first rendering information includes M first resolutions corresponding to M regions of each frame image, the i-th region is adjacent to the i+1-th region, the first resolution corresponding to the i-th region is greater than the first resolution corresponding to the i+1-th region, 1≤i≤M-1, M≥3, The second rendering information includes a second resolution corresponding to a transition area between a first area of ​​the previous frame image and a second area of ​​the next frame image, the first area includes other areas except the Mth area in the previous frame image, the second area includes other areas except the Mth area in the next frame image, and the second resolution is between an average resolution of the first area and an average resolution of the second area or between a minimum value of the first resolution corresponding to the first area and a minimum value of the first resolution corresponding to the second area; Acquire each frame of image according to its image identifier; Rendering the N frames of images according to the first rendering information to obtain N frames of rendered images, including: rendering the M regions of each frame of image so that the M regions of each frame of image have one-to-one corresponding M first resolutions; Transmitting the N frames of rendered images to the VR terminal; Performing color rendering on the transition area according to the second rendering information to obtain transition content with the second resolution; The transition content is transmitted to the VR terminal so that the VR terminal displays the transition content between the previous frame image and the next frame image.

2. The method according to claim 1, wherein: The j+1th region surrounds the jth region, where j is a value from 1 to M-1.

3. The method according to claim 2, wherein: The i+1th region surrounds the ith region.

4. The method according to claim 1, wherein: The first rendering information further includes size information and position information of the M regions of each frame image, and the rendering of the M regions of each frame image includes: Determine the M regions of each frame image according to the size information and position information of the M regions of the frame image; The frame image is rendered according to the M first resolutions corresponding to the M regions of each frame image, so that the M regions of the frame image have the M first resolutions corresponding to each other.

5. The method according to claim 1, wherein: The transmitting the N frames of rendered images to the VR terminal comprises: The N frames of rendered images are compressed to obtain at least one compressed image, and each compressed image is transmitted to the VR terminal, wherein each compressed image is obtained by compressing at least two consecutive frames of rendered images, and the at least two frames of rendered images in each compressed image are decompressed by the VR terminal and displayed frame by frame.

6. An image processing method, comprising: Receive rendering information and image identifiers of N frames of images sent by a virtual reality VR terminal, wherein the N frames of images include Q groups of images, each group of images includes an adjacent previous frame of image and a next frame of image, 1≤Q≤N-1, N≥2, the rendering information includes first rendering information, the first rendering information includes M first resolutions corresponding to M regions in each frame of image, the first region is adjacent to the Mth region, the first resolution corresponding to the first region is greater than the first resolution corresponding to the Mth region, and M=2, The rendering information further includes second rendering information, the second rendering information including a second resolution corresponding to a transition area between a first area of ​​the previous frame image and a second area of ​​the next frame image, the first area includes other areas except the Mth area in the previous frame image, the second area includes other areas except the Mth area in the next frame image, and the second resolution is between an average resolution of the first area and an average resolution of the second area or between a minimum value of the first resolution corresponding to the first area and a minimum value of the first resolution corresponding to the second area; Acquire each frame of image according to its image identifier; Rendering the N frames of images according to the first rendering information to obtain N frames of rendered images, including: rendering the M regions of each frame of image so that the M regions of each frame of image have one-to-one corresponding M first resolutions; Compressing the N frames of rendered images to obtain at least one compressed image, and transmitting each compressed image to the VR terminal, wherein each compressed image is obtained by compressing at least two consecutive frames of rendered images, and the at least two frames of rendered images in each compressed image are decompressed by the VR terminal and displayed frame by frame; Performing color rendering on the transition area according to the second rendering information to obtain transition content with the second resolution; The transition content is transmitted to the VR terminal so that the VR terminal displays the transition content between the previous frame image and the next frame image.

7. The method according to claim 5 or 6, wherein: Any compressed image is obtained as follows: Encoding M regions in each of the at least two rendered images in the any one compressed image based on M different encoding levels, wherein the encoding level corresponding to the i-th region is higher than the encoding level corresponding to the i+1-th region; The encoded at least two frames of rendered images are compressed to obtain any one compressed image.

8. The method according to claim 7, wherein: The at least one compressed image includes a first compressed image and a second compressed image, the N frames of rendered images include a continuous multi-frame rendered image, and the multi-frame rendered image is composed of at least two continuous frames of the first rendered image and at least two continuous frames of the second rendered image; The step of compressing the N frames of rendered images to obtain at least one compressed image, and transmitting each compressed image to the VR terminal comprises: Compressing the at least two frames of first rendered images to obtain the first compressed image; After obtaining the first compressed image, the first compressed image is transmitted to the VR terminal, and the at least two frames of the second rendered image are compressed to obtain the second compressed image, wherein the sum of the time for transmitting the first compressed image and the time for the VR terminal to display the first compressed image is greater than or equal to the sum of the time for obtaining the second compressed image and the time for transmitting the second compressed image.

9. The method according to claim 5 or 6, wherein: Each compressed image has the same capacity as one frame of rendered image.

10. The method according to claim 5 or 6, wherein: Each compressed image is obtained by compressing three consecutive frames of rendered images.

11. The method according to any one of claims 1 to 6, wherein: The N frames of images are rendered in parallel.

12. The method according to any one of claims 1 to 6, wherein: The M regions of each frame image are rendered in parallel.

13. The method according to claim 5 or 6, wherein: Each compressed image carries a quality of service (QoS) tag, and transmitting each compressed image to the VR terminal includes: Each compressed image is transmitted to the VR terminal using a network having a transmission quality corresponding to the QoS label.

14. An image processing method, comprising: Acquire historical viewing information of each frame of N frames of images, where the N frames of images include Q groups of images, each group of images includes an adjacent previous frame of image and a next frame of image, 1≤Q≤N-1, N≥2; Determine M regions in each frame of image based on historical viewing information of the frame of image, where the i-th region is adjacent to the i+1-th region, 1≤i≤M-1, M≥3; Determine first rendering information, where the first rendering information includes M first resolutions corresponding to M regions of each frame of image, where the first resolution corresponding to the i-th region is greater than the first resolution corresponding to the i+1-th region; Determine second rendering information, the second rendering information including a second resolution corresponding to a transition area between a first area of ​​the previous frame image and a second area of ​​the next frame image, the first area including other areas except the M-th area in the previous frame image, the second area including other areas except the M-th area in the next frame image, and the second resolution is between an average resolution of the first area and an average resolution of the second area or between a minimum value of the first resolution corresponding to the first area and a minimum value of the first resolution corresponding to the second area; sending the image identifiers of the N frames of images and the rendering information including the first rendering information and the second rendering information to a cloud server, so that the cloud server renders the N frames of images according to the first rendering information and the image identifiers of the N frames of images and performs color rendering on the transition area according to the second rendering information to obtain transition content with the second resolution; The transition content is displayed between the previous frame image and the next frame image.

15. The method according to claim 14, wherein: The first rendering information also includes size information and position information of the M regions of each frame image.

16. The method according to claim 14, wherein: N≥2, the determining of the M regions in each frame of image based on the historical viewing information of the frame of image comprises: Determine, based on historical viewing information of each frame of image, M regions of a first frame of image and M initial regions of each frame of image in N-1 frames of image among the N frames of image except the first frame of image; Get historical eye tracking data; Based on the centers of the other regions except the Mth region in the kth frame image and the historical eye tracking data, adjusting the positions of the other initial regions except the Mth initial region in the k+1th frame image, 1≤k≤N-1; The Mth initial region in the k+1th frame image and other adjusted initial regions are determined as the M regions of the k+1th frame image.

17. The method according to claim 16, wherein: The adjusting the positions of the other initial regions except the Mth initial region in the k+1th frame image comprises: Predicting the viewpoint of the k+1th frame image based on the centers of other regions except the Mth region in the kth frame image and the historical eye tracking data; The positions of the other initial regions are adjusted so that the centers of the adjusted other initial regions coincide with the viewpoint.

18. An image processing device, comprising: The receiving module is configured to receive rendering information and image identifiers of N frames of images sent by a virtual reality VR terminal, wherein the N frames of images include Q groups of images, each group of images includes an adjacent previous frame of image and a next frame of image, 1≤Q≤N-1, N≥2, and the rendering information includes first rendering information and second rendering information, wherein, The first rendering information includes M first resolutions corresponding to M regions of each frame image, the i-th region is adjacent to the i+1-th region, the first resolution corresponding to the i-th region is greater than the first resolution corresponding to the i+1-th region, 1≤i≤M-1, M≥3, The second rendering information includes a second resolution corresponding to a transition area between a first area of ​​the previous frame image and a second area of ​​the next frame image, the first area includes other areas except the Mth area in the previous frame image, the second area includes other areas except the Mth area in the next frame image, and the second resolution is between an average resolution of the first area and an average resolution of the second area or between a minimum value of the first resolution corresponding to the first area and a minimum value of the first resolution corresponding to the second area; An acquisition module is configured to acquire each frame of image according to the image identifier of the frame of image; a rendering module configured to render the N frames of images according to the first rendering information to obtain N frames of rendered images, including: rendering the M regions of each frame of image so that the M regions of each frame of image have one-to-one corresponding M first resolutions, and performing color rendering on the transition region according to the second rendering information to obtain transition content with the second resolution; The transmission module is configured to transmit the N frames of rendered images and the transition content to the VR terminal so that the VR terminal displays the transition content between the previous frame image and the next frame image.

19. An image processing device, comprising: A receiving module is configured to receive rendering information and image identifiers of N frames of images sent by a virtual reality VR terminal, wherein the N frames of images include Q groups of images, each group of images includes adjacent previous and next frames of images, 1≤Q≤N-1, N≥2, the rendering information includes first rendering information, the first rendering information includes M first resolutions corresponding to M regions in each frame of the image, the first region is adjacent to the Mth region, the first resolution corresponding to the first region is greater than the first resolution corresponding to the Mth region, M=2, The rendering information further includes second rendering information, the second rendering information including a second resolution corresponding to a transition area between a first area of ​​the previous frame image and a second area of ​​the next frame image, the first area includes other areas except the Mth area in the previous frame image, the second area includes other areas except the Mth area in the next frame image, and the second resolution is between an average resolution of the first area and an average resolution of the second area or between a minimum value of the first resolution corresponding to the first area and a minimum value of the first resolution corresponding to the second area; An acquisition module is configured to acquire each frame of image according to the image identifier of the frame of image; a rendering module configured to render the N frames of images according to the first rendering information to obtain N frames of rendered images, including: rendering the M regions of each frame of image so that the M regions of each frame of image have one-to-one corresponding M first resolutions, and performing color rendering on the transition region according to the second rendering information to obtain transition content with the second resolution; A transmission module is configured to compress the N frames of rendered images to obtain at least one compressed image, and transmit each compressed image and the transition content to the VR terminal so that the VR terminal displays the transition content between the previous frame image and the next frame image, wherein each compressed image is obtained by compressing at least two consecutive frames of rendered images, and the at least two frames of rendered images in each compressed image are decompressed by the VR terminal and displayed frame by frame.

20. An image processing device, comprising: Memory; as well as A processor coupled to the memory, configured to execute the method of any one of claims 1-13 based on instructions stored in the memory.

21. An image processing device, comprising: an acquisition module configured to acquire historical viewing information of each frame of N frames of images, wherein the N frames of images include Q groups of images, each group of images includes an adjacent previous frame of image and a next frame of image, 1≤Q≤N-1, N≥2; A first determination module is configured to determine M regions in each frame of image based on historical viewing information of the frame of image, where the i-th region is adjacent to the i+1-th region, 1≤i≤M-1, and M≥3; A second determining module is configured to determine first rendering information, wherein the first rendering information includes M first resolutions corresponding to M regions of each frame of image, and the first resolution corresponding to the i-th region is greater than the first resolution corresponding to the i+1-th region; The image processing device is configured to determine second rendering information, the second rendering information including a second resolution corresponding to a transition area between a first area of ​​the previous frame image and a second area of ​​the next frame image, the first area including other areas except the Mth area in the previous frame image, the second area including other areas except the Mth area in the next frame image, and the second resolution being between an average resolution of the first area and an average resolution of the second area or between a minimum value of the first resolution corresponding to the first area and a minimum value of the first resolution corresponding to the second area; The image processing device further comprises: a sending module configured to send the image identifiers of the N frames of images and the rendering information including the first rendering information and the second rendering information to a cloud server, so that the cloud server renders the N frames of images according to the first rendering information and the image identifiers of the N frames of images and performs color rendering on the transition area according to the second rendering information; The image processing device is configured to receive transition content having the second resolution obtained by color rendering, and display the transition content between the previous frame image and the next frame image.

22. An image processing device, comprising: Memory; as well as A processor coupled to the memory is configured to execute the method of any one of claims 14-17 based on instructions stored in the memory.

23. A cloud server, comprising: An image processing device as claimed in any one of claims 18 to 20.

24. A VR terminal, comprising: The image processing device according to claim 21 or 22.

25. An image processing system, comprising: The cloud server as claimed in claim 23; as well as The VR terminal as described in claim 24.

26. A computer-readable storage medium comprising computer program instructions, wherein: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 17 is implemented.

27. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 17 is implemented.

Images