Image processing method, device, equipment and readable storage medium

Abstract

The present invention discloses an image processing method, device, equipment and readable storage medium, which relates to the field of image processing technology to solve the problem of resource waste. The method includes: obtaining a VR frame image to be rendered; searching the first-level cache to determine whether there is a first cache image, and the similarity between the first cache image and the VR frame image to be rendered meets the first preset requirement; when the first-level cache does not have the first cache image, obtaining a second cache image from the second-level cache, and the similarity between the second cache image and the VR frame image to be rendered meets the second preset requirement; obtaining a difference image according to the VR frame image to be rendered and the second cache image; sending the difference image to the terminal; wherein the first-level cache and the second-level cache are respectively used to store the rendered VR frame image. The embodiment of the present invention can save resources of the edge cloud.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device, equipment and readable storage medium. Background Art

[0002] When mobile terminal load cloud technology is applied to VR (Virtual Reality) applications, services will be provided to each user independently. That is, each VR frame of the user is rendered separately by the edge cloud according to display requirements or fixed frame rate. For example, if the screen refresh rate is 60FPS, the edge cloud must ensure that one frame is rendered within 16.67ms and fully transmitted to the mobile terminal.

[0003] However, due to the limited computing power of the edge cloud, when the number of users increases, the existing solution will cause excessive load on the edge cloud, resulting in a waste of its resources. Summary of the invention

[0004] Embodiments of the present invention provide an image processing method, apparatus, device and readable storage medium to solve the problem of resource waste.

[0005] In a first aspect, an embodiment of the present invention provides an image processing method, which is applied to an edge cloud, including:

[0006] Get the VR frame image to be rendered;

[0007] Searching the first-level cache to determine whether there is a first cache image, wherein the similarity between the first cache image and the VR frame image to be rendered meets a first preset requirement;

[0008] When the first-level cache does not have the first cache image, obtaining a second cache image from the second-level cache, and the similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement;

[0009] Obtaining a difference image according to the VR frame image to be rendered and the second cache image;

[0010] Sending the difference image to a terminal;

[0011] The first-level cache and the second-level cache are respectively used to store rendered VR frame images.

[0012] Wherein, obtaining the second cache image from the second-level cache includes:

[0013] Obtaining the target camera pose of the VR frame image to be rendered;

[0014] Searching the second-level cache, and obtaining the third cache image from the second-level cache;

[0015] The second cache image is obtained according to the third cache image.

[0016] The step of obtaining the second cache image according to the third cache image includes:

[0017] If the distortion distance between the camera pose of the third cache image and the target camera pose is 0, using the third cache image as the second cache image;

[0018] If the distortion distance between the camera pose of the third cache image and the target camera pose is greater than 0 and less than or equal to a preset threshold, the third cache image is subjected to image distortion to obtain the second cache image.

[0019] The step of obtaining a difference image according to the VR frame image to be rendered and the second cache image includes:

[0020] The pixel value of the difference image in the i-th channel is obtained as follows:

[0021]

[0022] Among them, I delta (i) represents the i channel value of the difference image, I l (i) represents the i channel value of the VR frame image to be rendered, I c (i) represents the i channel value of the second cache image, i=Y, U, V.

[0023] Wherein, the method further comprises:

[0024] In a case where the first-level cache has the first cache image, the first cache image is sent to the terminal.

[0025] Wherein, the method further comprises:

[0026] In a case where the second-level cache does not have the second cache image, rendering the VR frame image to be rendered, and sending the rendered VR frame image to the terminal;

[0027] The rendered VR frame image is stored in the second-level cache.

[0028] In a second aspect, an embodiment of the present invention further provides an image processing method, applied to a terminal, comprising:

[0029] Acquire a difference image from the edge cloud, wherein the difference image is obtained by the edge cloud according to the VR frame image to be rendered and the rendered VR frame image cached by the edge cloud;

[0030] Get the VR frame image at the current moment;

[0031] A VR frame image to be displayed is obtained according to the VR frame image at the current moment and the difference image.

[0032] The step of obtaining the VR frame image to be displayed according to the VR frame image at the current moment and the difference image includes:

[0033] Determine whether a distortion distance between a camera pose of the VR frame image at the current moment and a camera pose of the VR frame image to be rendered is greater than 0;

[0034] If it is greater than 0, the difference image is distorted;

[0035] A VR frame image to be displayed is obtained according to the VR frame image at the current moment and the distorted difference image.

[0036] The step of obtaining the VR frame image to be displayed according to the VR frame image at the current moment and the distorted difference image includes:

[0037] The pixel value of the VR frame image to be displayed in the i-th channel is obtained as follows:

[0038] I new (i) = min[2×(I delta (i)-127)+I d (i), 255)

[0039] Among them, I new (i) represents the i channel value of the VR frame image to be displayed, I delta (i) represents the i channel value of the difference image, I d (i) represents the i channel value of the VR frame image at the current moment, i=Y, U, V.

[0040] In a third aspect, an embodiment of the present invention further provides an image processing device, applied to an edge cloud, comprising:

[0041] A first acquisition module, used to acquire a VR frame image to be rendered;

[0042] A first determination module is used to search the first-level cache to determine whether there is a first cache image, and the similarity between the first cache image and the VR frame image to be rendered meets a first preset requirement;

[0043] A second acquisition module, configured to acquire a second cache image from a second-level cache when the first-level cache does not have the first cache image, wherein a similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement;

[0044] A third acquisition module, configured to obtain a difference image according to the VR frame image to be rendered and the second cache image;

[0045] A first sending module, used for sending the difference image to a terminal;

[0046] The first-level cache and the second-level cache are respectively used to store rendered VR frame images.

[0047] Wherein, the second acquisition module includes:

[0048] A first acquisition submodule is used to acquire the target camera pose of the VR frame image to be rendered;

[0049] A first search submodule, used for searching the second-level cache and obtaining a third cache image from the second-level cache;

[0050] The second acquisition submodule is used to obtain the second cache image according to the third cache image.

[0051] Wherein, the second acquisition submodule includes:

[0052] a first acquisition unit, configured to use the third cache image as the second cache image if the distortion distance between the camera pose of the third cache image and the target camera pose is 0;

[0053] The second acquisition unit is used to perform image distortion on the third cache image to obtain the second cache image if the distortion distance between the camera pose of the third cache image and the target camera pose is greater than 0 and less than or equal to a preset threshold.

[0054] Wherein, the third acquisition module is used for:

[0055] The pixel value of the difference image in the i-th channel is obtained as follows:

[0056]

[0057] Among them, I delta (i) represents the i channel value of the difference image, I l (i) represents the i channel value of the VR frame image to be rendered, I c (i) represents the i channel value of the second cache image, i=Y, U, V.

[0058] Wherein, the device further comprises:

[0059] The second sending module is configured to send the first cache image to the terminal when the first-level cache has the first cache image.

[0060] In a fourth aspect, an embodiment of the present invention further provides an image processing device, applied to a terminal, comprising:

[0061] A first acquisition module is used to acquire a difference image from an edge cloud, wherein the difference image is obtained by the edge cloud according to a VR frame image to be rendered and a rendered VR frame image cached by the edge cloud;

[0062] The second acquisition module is used to acquire the VR frame image at the current moment;

[0063] The third acquisition module is used to obtain the VR frame image to be displayed according to the VR frame image at the current moment and the difference image.

[0064] Wherein, the third acquisition module includes:

[0065] A first determination submodule is used to determine whether a distortion distance between a camera pose of the VR frame image at the current moment and a camera pose of the VR frame image to be rendered is greater than 0;

[0066] A first processing submodule, configured to perform image distortion on the difference image if the difference image is greater than 0;

[0067] The first acquisition submodule is used to obtain the VR frame image to be displayed according to the VR frame image at the current moment and the distorted difference image.

[0068] The third acquisition module is used to obtain the pixel value of the VR frame image to be displayed in the i-th channel in the following manner:

[0069] I new (i) = min[2×(I delta (i)-127)+I d (i), 255)

[0070] Among them, I new (i) represents the i channel value of the VR frame image to be displayed, I delta (i) represents the i channel value of the difference image, I d (i) represents the i channel value of the VR frame image at the current moment, i=Y, U, V.

[0071] In a fifth aspect, an embodiment of the present invention further provides an image processing device, applied to an edge cloud, comprising: a processor and a transceiver;

[0072] Wherein, the processor is used for:

[0073] Get the VR frame image to be rendered;

[0074] Searching the first-level cache to determine whether there is a first cache image, wherein the similarity between the first cache image and the VR frame image to be rendered meets a first preset requirement;

[0075] When the first-level cache does not have the first cache image, obtaining a second cache image from the second-level cache, and the similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement;

[0076] Obtaining a difference image according to the VR frame image to be rendered and the second cache image;

[0077] Sending the difference image to a terminal;

[0078] The first-level cache and the second-level cache are respectively used to store rendered VR frame images.

[0079] Wherein, the processor is further used for:

[0080] Obtaining the target camera pose of the VR frame image to be rendered;

[0081] Searching the second-level cache, and obtaining the third cache image from the second-level cache;

[0082] The second cache image is obtained according to the third cache image.

[0083] Wherein, the processor is further used for:

[0084] If the distortion distance between the camera pose of the third cache image and the target camera pose is 0, using the third cache image as the second cache image;

[0085] If the distortion distance between the camera pose of the third cache image and the target camera pose is greater than 0 and less than or equal to a preset threshold, the third cache image is subjected to image distortion to obtain the second cache image.

[0086] Wherein, the processor is further used for:

[0087] The pixel value of the difference image in the i-th channel is obtained as follows:

[0088]

[0089] Among them, I delta (i) represents the i channel value of the difference image, I l (i) represents the i channel value of the VR frame image to be rendered, I c(i) represents the i channel value of the second cache image, i=Y, U, V.

[0090] Wherein, the processor is further used for:

[0091] In a case where the first-level cache has the first cache image, the first cache image is sent to the terminal.

[0092] Wherein, the processor is further used for:

[0093] In a case where the second-level cache does not have the second cache image, rendering the VR frame image to be rendered, and sending the rendered VR frame image to the terminal;

[0094] The rendered VR frame image is stored in the second-level cache.

[0095] In a fifth aspect, an embodiment of the present invention further provides an image processing device, applied to a terminal, comprising: a processor and a transceiver;

[0096] Wherein, the processor is used for:

[0097] Acquire a difference image from the edge cloud, wherein the difference image is obtained by the edge cloud according to the VR frame image to be rendered and the rendered VR frame image cached by the edge cloud;

[0098] Get the VR frame image at the current moment;

[0099] A VR frame image to be displayed is obtained according to the VR frame image at the current moment and the difference image.

[0100] Wherein, the processor is further used for:

[0101] The obtaining the VR frame image to be displayed according to the VR frame image at the current moment and the difference image comprises:

[0102] Determine whether a distortion distance between a camera pose of the VR frame image at the current moment and a camera pose of the VR frame image to be rendered is greater than 0;

[0103] If it is greater than 0, the difference image is distorted;

[0104] A VR frame image to be displayed is obtained according to the VR frame image at the current moment and the distorted difference image.

[0105] The processor is further configured to obtain the pixel value of the VR frame image to be displayed in the i-th channel in the following manner:

[0106] I new (i) = min[2×(Idelta (i)-127)+I d (i), 255)

[0107] Among them, I new (i) represents the i channel value of the VR frame image to be displayed, I delta (i) represents the i channel value of the difference image, I d (i) represents the i channel value of the VR frame image at the current moment, i=Y, U, V.

[0108] In a seventh aspect, an embodiment of the present invention further provides a readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps in the image processing method as described above are implemented.

[0109] In an eighth aspect, an embodiment of the present invention further provides a communication device, comprising: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps of the image processing method as described above.

[0110] In an embodiment of the present invention, when a VR frame image needs to be rendered, a search is performed in the first-level cache and second-level storage of the edge cloud to obtain a rendered VR frame image whose similarity with the VR frame image meets preset requirements, thereby avoiding unnecessary computational waste of VR frame rendering on the edge cloud and saving resources of the edge cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0111] Figure 1 is one of the flow charts of the image processing method provided by the embodiment of the present invention;

[0112] Figure 2 is a schematic diagram of a storage structure provided by an embodiment of the present invention;

[0113] Figure 3 This is the second flowchart of the image processing method provided by the embodiment of the present invention;

[0114] Figure 4 is a processing schematic diagram provided by an embodiment of the present invention;

[0115] Figure 5 is one of the structural diagrams of the image processing device provided by the embodiment of the present invention;

[0116] Figure 6 is a second structural diagram of the image processing device provided by an embodiment of the present invention;

[0117] Figure 7 is a third structural diagram of the image processing device provided by an embodiment of the present invention;

[0118] Figure 8 This is the fourth structural diagram of the image processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0119] In the embodiments of the present invention, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0120] In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0121] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0122] See also Figure 1 , Figure 1 is a flow chart of an image processing method provided by an embodiment of the present invention, which is applied to edge cloud, such as Figure 1 As shown, the following steps are included:

[0123] Step 101: Obtain a VR frame image to be rendered.

[0124] When VR frame image rendering is required, the edge cloud can obtain the VR frame image to be rendered from the terminal.

[0125] Step 102: search the first-level cache to determine whether there is a first cache image, and the similarity between the first cache image and the VR frame image to be rendered meets the first preset requirement.

[0126] like Figure 2 As shown, in an embodiment of the present invention, two levels of storage are established in the edge cloud, including a first-level cache and a second-level storage. The first-level cache and the second-level cache are respectively used to store rendered VR frame images.

[0127] Specifically, the first-level cache can be used to store the rendered VR frame images corresponding to a certain terminal, and the second-level storage is used to store the rendered VR frame images of all terminals. One or more terminals can correspond to the same or more first-level caches.

[0128] In the process of delivering the rendered images from the second-level cache to each terminal, inter-process communication (IPC) will inevitably occur. In order to minimize the operating overhead of such a system, a small local cache (i.e., the first-level cache) is created and maintained for each terminal, which stores the locally rendered images for faster reuse. Generally, according to the VR frame rate of 60fps, the cache in the second-level cache library is designed to store images within 5s, that is, 300 frames of images.

[0129] When a request from a terminal is received, the first-level cache corresponding to the terminal may be searched first. For example, the first-level cache corresponding to the terminal may be searched according to the representation of the terminal. Specifically, the first-level cache may be searched according to the target camera pose of the VR frame image to be rendered to determine whether there is an image in the first-level cache whose similarity with the first cache image and the VR frame image to be rendered meets the first preset requirement. For example, the first preset requirement may be that the distance between the camera poses of the images is less than a preset value (which can be set as needed), or that the similarity of the background is greater than a preset value (which can be set as needed), etc. If yes, the first cache image may be directly sent to the terminal. If not, the second-level cache needs to be searched.

[0130] Step 103: When the first-level cache does not have the first cache image, obtain a second cache image from the second-level cache, and the similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement.

[0131] For example, the second preset requirement may be that the distance between the camera postures of the images is less than a preset value (which can be set as needed), etc.

[0132] Specifically, in this step, the target camera pose of the VR frame image to be rendered can be obtained, and then the second-level cache is searched, and the third cache image is obtained from the second-level cache. Finally, the second cache image is obtained according to the third cache image.

[0133] If the distortion distance between the camera pose of the third cache image and the target camera pose is 0, it can be considered that the third cache image and the second cache image completely match or the similarity is 100%, then the third cache image can be used as the second cache image. If the distortion distance between the camera pose of the third cache image and the target camera pose is greater than 0 and less than or equal to a preset threshold (wherein the threshold can be set as required), the third cache image is distorted to obtain the second cache image.

[0134] In specific applications, choosing a larger threshold can distort the image to a farther distance and save more calculations. In general, the preset threshold D threshold =5.0, that is, at a distance of 5m in the actual environment, the images before and after the distortion can be kept more than 40% repeated; the preset threshold value D threshold =2.0, the images before and after the distortion can be kept more than 60% repeated. In the embodiment of the present invention, a preset threshold D threshold =2.0.

[0135] Step 104: Obtain a difference image according to the VR frame image to be rendered and the second cache image.

[0136] In order to avoid transmitting the complete VR frame image, reduce rendering and encoding and decoding delays and save edge cloud transmission bandwidth, in an embodiment of the present invention, incremental images (or difference images) of two reference frames are transmitted. Among them, the two reference frames are: the VR frame image to be rendered and the second cache image. The difference image refers to the difference between the two reference frames. The difference image is transmitted through compression encoding such as H.264, H.265 technology, etc., and by adjusting the compression rate of the encoding, the network transmission bandwidth is saved while ensuring the picture quality. Specifically, if the VR frame image is transmitted with 8-bit color depth per channel, the pixel value of the difference image in the i-th channel is:

[0137]

[0138] Among them, I delta (i) represents the i channel value of the difference image, I l (i) represents the i channel value of the VR frame image to be rendered, I c (i) represents the i channel value of the second cache image, i=Y, U, V.

[0139] That is, the channel values ​​of the difference image in the three channels of Y, U, and V are calculated respectively according to the above formula, and its calculation result maps the positive and negative values ​​of the difference image to lighter and darker colors respectively.

[0140] Step 105: Send the difference image to the terminal.

[0141] During the sending process, operations such as encoding may be performed on the difference image.

[0142] In an embodiment of the present invention, when a VR frame image needs to be rendered, a search is performed in the first-level cache and second-level storage of the edge cloud to obtain a rendered VR frame image whose similarity with the VR frame image meets preset requirements, thereby avoiding unnecessary computational waste of VR frame rendering on the edge cloud and saving resources of the edge cloud.

[0143] See also Figure 3 , Figure 3 is a flowchart of an image processing method provided by an embodiment of the present invention, which is applied to a terminal, such as Figure 3 As shown, the following steps are included:

[0144] Step 301: Obtain a difference image from an edge cloud, wherein the difference image is obtained by the edge cloud based on a VR frame image to be rendered and a rendered VR frame image cached by the edge cloud.

[0145] The way in which the edge cloud obtains the difference image may refer to the description of the above embodiment. After obtaining the difference image, operations such as decoding may be performed on it.

[0146] Step 302: Obtain the VR frame image at the current moment.

[0147] The current moment may refer to a VR frame image obtained by the terminal when receiving the difference image.

[0148] Step 303: Obtain a VR frame image to be displayed according to the VR frame image at the current moment and the difference image.

[0149] Since the camera pose may change during the transmission of the difference image, in order to make the obtained image more in line with the needs of the user, in an embodiment of the present invention, it is determined whether the distortion distance between the camera pose of the VR frame image at the current moment and the camera pose of the VR frame image to be rendered is greater than 0. If it is greater than 0, the difference image is distorted so that it is consistent with the camera pose of the VR frame image at the current moment. If it is equal to 0, no distortion operation is performed. Afterwards, the VR frame image to be displayed is obtained based on the VR frame image at the current moment and the distorted difference image. Specific distortion methods may include ASW (Asynchronous Spacewarp), ATW (Asynchronous Timewarp) algorithms, etc.

[0150] The pixel value of the VR frame image to be displayed in the i-th channel can be obtained as follows:

[0151] I new (i) = min[2×(I delta (i)-127)+I d (i), 255)

[0152] Among them, I new (i) represents the i channel value of the VR frame image to be displayed, I delta (i) represents the i channel value of the difference image, I d(i) represents the i channel value of the VR frame image at the current moment, i=Y, U, V.

[0153] In an embodiment of the present invention, when a VR frame image needs to be rendered, a search is performed in the first-level cache and second-level storage of the edge cloud to obtain a rendered VR frame image whose similarity with the VR frame image meets preset requirements, thereby avoiding unnecessary computational waste of VR frame rendering on the edge cloud and saving resources of the edge cloud.

[0154] like Figure 4 As shown, in order to minimize latency, in an embodiment of the present invention, a parallel streaming processing mechanism is adopted. In addition, during rendering and image search, a left-eye and right-eye separation method is adopted. While rendering and searching for the image for the right eye (denoted as R0) of frame 0, the difference image of the left eye of frame 0 (L0) is encoded. During execution, in order to avoid processing pauses or idle resources due to the heterogeneous computing complexity of different stages, a request queue is maintained for each stage, and then the next task can be performed immediately without waiting.

[0155] It should be noted that image warping does not impose any restrictions on the target camera pose, allowing for perspective reprojection in the virtual world. However, the visual quality of image warping depends on the accuracy of measuring the user's pose in the virtual world, i.e., the user's movements should be accurately depicted and reflected. The accuracy of this pose measurement is inherently determined by the mobile HMD (Head Mounted Device) hardware, which implements real-time acquisition and tracking of the user's movements.

[0156] At the same time, in the embodiment of the present invention, the cache performance is optimized through the secondary cache mechanism, and has the advantages of centralized and distributed cache solutions. Therefore, in this way, the IPC inter-process communication during data transmission can be reduced, and the transmission bandwidth and transmission delay can be reduced by transmitting only the difference image.

[0157] The embodiment of the present invention also provides an image processing device, which is applied to edge cloud. Figure 5 , Figure 5 1 is a structural diagram of an image processing device provided by an embodiment of the present invention. Since the principle of solving the problem by the image processing device is similar to that of the image processing method in the embodiment of the present invention, the implementation of the image processing device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0158] like Figure 5 As shown, the image processing device 500 includes:

[0159] The first acquisition module 501 is used to acquire the VR frame image to be rendered; the first determination module 502 is used to search the first-level cache to determine whether there is a first cache image, and the similarity between the first cache image and the VR frame image to be rendered meets the first preset requirement; the second acquisition module 503 is used to acquire the second cache image from the second-level cache when the first-level cache does not have the first cache image, and the similarity between the second cache image and the VR frame image to be rendered meets the second preset requirement; the third acquisition module 504 is used to obtain a difference image based on the VR frame image to be rendered and the second cache image; the first sending module 505 is used to send the difference image to the terminal; wherein the first-level cache and the second-level cache are respectively used to store rendered VR frame images.

[0160] Wherein, the second acquisition module includes:

[0161] A first acquisition submodule is used to acquire the target camera pose of the VR frame image to be rendered;

[0162] A first search submodule, used for searching the second-level cache and obtaining a third cache image from the second-level cache;

[0163] The second acquisition submodule is used to obtain the second cache image according to the third cache image.

[0164] Wherein, the second acquisition submodule includes:

[0165] a first acquisition unit, configured to use the third cache image as the second cache image if the distortion distance between the camera pose of the third cache image and the target camera pose is 0;

[0166] The second acquisition unit is used to perform image distortion on the third cache image to obtain the second cache image if the distortion distance between the camera pose of the third cache image and the target camera pose is greater than 0 and less than or equal to a preset threshold.

[0167] Wherein, the third acquisition module is used for:

[0168] The pixel value of the difference image in the i-th channel is obtained as follows:

[0169]

[0170] Among them, I delta (i) represents the i channel value of the difference image, I l (i) represents the i channel value of the VR frame image to be rendered, I c (i) represents the i channel value of the second cache image, i=Y, U, V.

[0171] Wherein, the device further comprises:

[0172] The second sending module is configured to send the first cache image to the terminal when the first-level cache has the first cache image.

[0173] The device provided in the embodiment of the present invention can execute the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0174] The embodiment of the present invention also provides an image processing device, which is applied to a terminal. Figure 6 , Figure 6 1 is a structural diagram of an image processing device provided by an embodiment of the present invention. Since the principle of solving the problem by the image processing device is similar to that of the image processing method in the embodiment of the present invention, the implementation of the image processing device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0175] like Figure 6 As shown, the image processing device 600 includes:

[0176] The first acquisition module 601 is used to obtain a difference image from the edge cloud, wherein the difference image is obtained by the edge cloud based on the VR frame image to be rendered and the rendered VR frame image cached by the edge cloud; the second acquisition module 602 is used to obtain the VR frame image at the current moment; the third acquisition module 603 is used to obtain the VR frame image to be displayed based on the VR frame image at the current moment and the difference image.

[0177] Wherein, the third acquisition module includes:

[0178] A first determination submodule is used to determine whether a distortion distance between a camera pose of the VR frame image at the current moment and a camera pose of the VR frame image to be rendered is greater than 0;

[0179] A first processing submodule, configured to perform image distortion on the difference image if the difference image is greater than 0;

[0180] The first acquisition submodule is used to obtain the VR frame image to be displayed according to the VR frame image at the current moment and the distorted difference image.

[0181] The third acquisition module is used to obtain the pixel value of the VR frame image to be displayed in the i-th channel in the following manner:

[0182] I new (i) = min[2×(I delta (i)-127)+I d (i), 255)

[0183] Among them, I new (i) represents the i channel value of the VR frame image to be displayed, I delta (i) represents the i channel value of the difference image, I d (i) represents the i channel value of the VR frame image at the current moment, i=Y, U, V.

[0184] The device provided in the embodiment of the present invention can execute the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0185] The embodiment of the present invention also provides an image processing device, which is applied to edge cloud. Figure 7 , Figure 7 1 is a structural diagram of an image processing device provided by an embodiment of the present invention. Since the principle of solving the problem by the image processing device is similar to that of the image processing method in the embodiment of the present invention, the implementation of the image processing device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0186] like Figure 7 As shown, the image processing device 700 includes: a processor 701 and a transceiver 702 .

[0187] The processor 701 is used for:

[0188] Get the VR frame image to be rendered;

[0189] Searching the first-level cache to determine whether there is a first cache image, wherein the similarity between the first cache image and the VR frame image to be rendered meets a first preset requirement;

[0190] When the first-level cache does not have the first cache image, obtaining a second cache image from the second-level cache, and the similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement;

[0191] Obtaining a difference image according to the VR frame image to be rendered and the second cache image;

[0192] Sending the difference image to a terminal;

[0193] The first-level cache and the second-level cache are respectively used to store rendered VR frame images.

[0194] The processor 701 is further configured to:

[0195] Obtaining the target camera pose of the VR frame image to be rendered;

[0196] Searching the second-level cache, and obtaining the third cache image from the second-level cache;

[0197] The second cache image is obtained according to the third cache image.

[0198] The processor 701 is further configured to:

[0199] If the distortion distance between the camera pose of the third cache image and the target camera pose is 0, using the third cache image as the second cache image;

[0200] If the distortion distance between the camera pose of the third cache image and the target camera pose is greater than 0 and less than or equal to a preset threshold, the third cache image is subjected to image distortion to obtain the second cache image.

[0201] The processor 701 is further configured to:

[0202] The pixel value of the difference image in the i-th channel is obtained as follows:

[0203]

[0204] Among them, I delta (i) represents the i channel value of the difference image, I l (i) represents the i channel value of the VR frame image to be rendered, I c (i) represents the i channel value of the second cache image, i=Y, U, V.

[0205] The processor 701 is further configured to:

[0206] In a case where the first-level cache has the first cache image, the first cache image is sent to the terminal.

[0207] The processor 701 is further configured to:

[0208] In a case where the second-level cache does not have the second cache image, rendering the VR frame image to be rendered, and sending the rendered VR frame image to the terminal;

[0209] The rendered VR frame image is stored in the second-level cache.

[0210] The device provided in the embodiment of the present invention can execute the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0211] The embodiment of the present invention also provides an image processing device, which is applied to edge cloud. Figure 8 , Figure 81 is a structural diagram of an image processing device provided by an embodiment of the present invention. Since the principle of solving the problem by the image processing device is similar to that of the image processing method in the embodiment of the present invention, the implementation of the image processing device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0212] like Figure 8 As shown, the image processing device 800 includes: a processor 801 and a transceiver 802 .

[0213] The processor 801 is used for:

[0214] Acquire a difference image from the edge cloud, wherein the difference image is obtained by the edge cloud according to the VR frame image to be rendered and the rendered VR frame image cached by the edge cloud;

[0215] Get the VR frame image at the current moment;

[0216] A VR frame image to be displayed is obtained according to the VR frame image at the current moment and the difference image.

[0217] The processor 801 is further configured to:

[0218] The obtaining the VR frame image to be displayed according to the VR frame image at the current moment and the difference image comprises:

[0219] Determine whether a distortion distance between a camera pose of the VR frame image at the current moment and a camera pose of the VR frame image to be rendered is greater than 0;

[0220] If it is greater than 0, the difference image is distorted;

[0221] A VR frame image to be displayed is obtained according to the VR frame image at the current moment and the distorted difference image.

[0222] The processor 801 is further configured to obtain the pixel value of the VR frame image to be displayed in the i-th channel in the following manner:

[0223] I new (i) = min[2×(I delta (i)-127)+I d (i), 255)

[0224] Among them, I new (i) represents the i channel value of the VR frame image to be displayed, I delta (i) represents the i channel value of the difference image, I d (i) represents the i channel value of the VR frame image at the current moment, i=Y, U, V.

[0225] The device provided in the embodiment of the present invention can execute the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0226] An embodiment of the present invention further provides a communication device, comprising: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps of the image processing method described above.

[0227] The embodiment of the present invention further provides a readable storage medium, on which a program is stored. When the program is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid state drive (SSD)), etc.

[0228] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0229] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal (which can be a mobile phone, computer, server, air conditioner, or network equipment, etc.) to execute the methods described in each embodiment of the present invention.

[0230] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. An image processing method, applied to edge cloud, characterized in that: include: Obtain the virtual reality VR frame image to be rendered; Searching the first-level cache to determine whether there is a first cache image, wherein the similarity between the first cache image and the VR frame image to be rendered meets a first preset requirement; When the first-level cache does not have the first cache image, obtaining a second cache image from the second-level cache, and the similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement; Obtaining a difference image according to the VR frame image to be rendered and the second cache image; Sending the difference image to a terminal; The first-level cache and the second-level cache are respectively used to store rendered VR frame images; the first-level cache is used to store rendered VR frame images corresponding to a certain terminal, and the second-level cache is used to store rendered VR frame images of all terminals; The obtaining a difference image according to the VR frame image to be rendered and the second cache image includes: The pixel value of the difference image in the i-th channel is obtained as follows: Among them, I delta (i) represents the i channel value of the difference image, I l (i) represents the i channel value of the VR frame image to be rendered, I c (i) represents the i channel value of the second cache image, i=Y, U, V.

2. The method according to claim 1, characterized in that The obtaining of the second cache image from the second level cache comprises: Obtaining the target camera pose of the VR frame image to be rendered; Searching the second-level cache, and obtaining the third cache image from the second-level cache; The second cache image is obtained according to the third cache image.

3. The method according to claim 2, characterized in that The step of obtaining the second cache image according to the third cache image includes: If the distortion distance between the camera pose of the third cache image and the target camera pose is 0, using the third cache image as the second cache image; If the distortion distance between the camera pose of the third cache image and the target camera pose is greater than 0 and less than or equal to a preset threshold, the third cache image is subjected to image distortion to obtain the second cache image.

4. The method according to claim 1, characterized in that The method further comprises: In a case where the first-level cache has the first cache image, the first cache image is sent to the terminal.

5. The method according to claim 1, characterized in that The method further comprises: In a case where the second-level cache does not have the second cache image, rendering the VR frame image to be rendered, and sending the rendered VR frame image to the terminal; The rendered VR frame image is stored in the second-level cache.

6. An image processing method, applied to a terminal, characterized in that: include: Acquire a difference image from the edge cloud, wherein the difference image is obtained by the edge cloud according to the VR frame image to be rendered and the rendered VR frame image cached by the edge cloud; Get the VR frame image at the current moment; Obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the difference image; The obtaining the VR frame image to be displayed according to the VR frame image at the current moment and the difference image comprises: Determine whether a distortion distance between a camera pose of the VR frame image at the current moment and a camera pose of the VR frame image to be rendered is greater than 0; If it is greater than 0, the difference image is distorted; Obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the distorted difference image; The obtaining of a VR frame image to be displayed according to the VR frame image at the current moment and the distorted difference image comprises: The pixel value of the VR frame image to be displayed in the i-th channel is obtained as follows: IN new (i)=min[2×(I delta (i)-127)+I d (i), 255) Among them, I new (i) represents the i channel value of the VR frame image to be displayed, I delta (i) represents the i channel value of the difference image, I d (i) represents the i channel value of the VR frame image at the current moment, i=Y, U, V.

7. An image processing device, applied to edge cloud, characterized in that: include: A first acquisition module, used to acquire a VR frame image to be rendered; A first determination module is used to search the first-level cache to determine whether there is a first cache image, and the similarity between the first cache image and the VR frame image to be rendered meets a first preset requirement; A second acquisition module, configured to acquire a second cache image from a second-level cache when the first-level cache does not have the first cache image, wherein a similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement; A third acquisition module, configured to obtain a difference image according to the VR frame image to be rendered and the second cache image; A first sending module, used for sending the difference image to a terminal; The first-level cache and the second-level cache are respectively used to store rendered VR frame images; the first-level cache is used to store rendered VR frame images corresponding to a certain terminal, and the second-level cache is used to store rendered VR frame images of all terminals; The third acquisition module is used for: The pixel value of the difference image in the i-th channel is obtained as follows: Among them, I delta (i) represents the i channel value of the difference image, I l (i) represents the i channel value of the VR frame image to be rendered, I c (i) represents the i channel value of the second cache image, i=Y, U, V.

8. An image processing device, applied to a terminal, characterized in that: include: A first acquisition module is used to acquire a difference image from an edge cloud, wherein the difference image is obtained by the edge cloud according to a VR frame image to be rendered and a rendered VR frame image cached by the edge cloud; The second acquisition module is used to acquire the VR frame image at the current moment; A third acquisition module, configured to obtain a VR frame image to be displayed according to the VR frame image at the current moment and the difference image; Wherein, the third acquisition module includes: A first determination submodule is used to determine whether a distortion distance between a camera pose of the VR frame image at the current moment and a camera pose of the VR frame image to be rendered is greater than 0; A first processing submodule, configured to perform image distortion on the difference image if the difference image is greater than 0; A first acquisition submodule, configured to obtain a VR frame image to be displayed according to the VR frame image at the current moment and the distorted difference image; The first acquisition submodule is specifically used to obtain the pixel value of the VR frame image to be displayed in the i-th channel in the following manner: IN new (i)=min[2×(I delta (i)-127)+I d (i), 255) Among them, I new (i) represents the i channel value of the VR frame image to be displayed, I delta (i) represents the i channel value of the difference image, I d (i) represents the i channel value of the VR frame image at the current moment, i=Y, U, V.

9. An image processing device, applied to edge cloud, characterized in that: include: processors and transceivers; Wherein, the processor is used for: Get the VR frame image to be rendered; Searching the first-level cache to determine whether there is a first cache image, wherein the similarity between the first cache image and the VR frame image to be rendered meets a first preset requirement; When the first-level cache does not have the first cache image, obtaining a second cache image from the second-level cache, and the similarity between the second cache image and the VR frame image to be rendered meets a second preset requirement; Obtaining a difference image according to the VR frame image to be rendered and the second cache image; Sending the difference image to a terminal; The first-level cache and the second-level cache are respectively used to store rendered VR frame images; the first-level cache is used to store rendered VR frame images corresponding to a certain terminal, and the second-level cache is used to store rendered VR frame images of all terminals; The processor is further used to obtain the pixel value of the difference image in the i-th channel in the following manner: Among them, I delta (i) represents the i channel value of the difference image, I l (i) represents the i channel value of the VR frame image to be rendered, I c (i) represents the i channel value of the second cache image, i=Y, U, V.

10. An image processing device, applied to a terminal, characterized in that: include: processors and transceivers; Wherein, the processor is used for: Acquire a difference image from the edge cloud, wherein the difference image is obtained by the edge cloud according to the VR frame image to be rendered and the rendered VR frame image cached by the edge cloud; Get the VR frame image at the current moment; Obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the difference image; The processor is further configured to: The obtaining the VR frame image to be displayed according to the VR frame image at the current moment and the difference image comprises: Determine whether a distortion distance between a camera pose of the VR frame image at the current moment and a camera pose of the VR frame image to be rendered is greater than 0; If it is greater than 0, the difference image is distorted; Obtaining a VR frame image to be displayed according to the VR frame image at the current moment and the distorted difference image; The processor is further configured to obtain the pixel value of the VR frame image to be displayed in the i-th channel in the following manner: IN new (i)=min[2×(I delta (i)-127)+I d (i), 255) Among them, I new (i) represents the i channel value of the VR frame image to be displayed, I delta (i) represents the i channel value of the difference image, I d (i) represents the i channel value of the VR frame image at the current moment, i=Y, U, V.

11. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

12. A communication device comprising: A transceiver, a memory, a processor and a program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps of the image processing method as described in any one of claims 1 to 6.

Images