Image processing method and apparatus, display apparatus and virtual reality display system
By determining the user's gaze area and calculating image similarity in VR technology, and transmitting only images with high similarity and performing image fusion processing, the network bandwidth pressure caused by the limited processing power of VR devices is solved, thus improving the user experience.
Patent Information
- Application Number
- PCT/CN2024/088656
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-17
- Filing Date
- 2024-04-18
- Publication Date
- 2024-11-21
AI Technical Summary
In existing VR technologies, due to the limitations of VR device processors, high-resolution image transmission puts a heavy strain on network bandwidth, making it difficult to meet real-time requirements and easily leading to latency and stuttering, which affects the user experience.
By identifying the user's gaze area, reducing image resolution, and calculating image similarity, only images with similarity higher than a threshold are transmitted. Combined with image fusion processing, this ensures the clarity and information integrity of the user's region of interest, while reducing network bandwidth pressure.
It improves the real-time performance of image transmission, reduces network bandwidth pressure, avoids latency and lag, and enhances the user experience.
Smart Images

Figure CN2024088656_21112024_PF_FP_ABST
Abstract
Description
Image processing method and device, display device, and virtual reality display system
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This disclosure claims priority from application number: 202310559344.5, application date: May 17, 2023, and invention title: Image processing method and device, display device, virtual reality display system. The entire contents of this Chinese patent application are incorporated herein by reference. Technical Field
[0003] The present disclosure relates to the field of image processing technology, and in particular to an image processing method and device, a display device, and a virtual reality display system. Background Art
[0004] With the development of virtual reality (VR) technology, users' requirements for immersion and comfort brought by VR are constantly increasing, which brings great challenges to image processing technology.
[0005] For large-scale VR applications, due to the limited processing power of VR device processors, a split approach—using high-performance servers for complex image processing—can achieve reliable and rapid data processing. However, VR images currently transmit full HD images, and with the increase in video refresh rates and screen clarity, the amount of data transmitted over the network continues to increase, placing significant pressure on network bandwidth, making it difficult to meet real-time requirements. This can easily lead to delays and lags, impacting the user experience.
[0006] Summary of the Invention
[0007] According to a first aspect of the present disclosure, an image processing method is provided, the method comprising: determining a corresponding first image from an original image of a current frame based on a user's gaze area; and determining a second image corresponding to the current frame based on the original image of the current frame, the second image having a lower resolution than the first image; determining a similarity between the second image corresponding to the current frame and the second image corresponding to the previous frame; and transmitting a target image based on the similarity; wherein, when the similarity is greater than or equal to a preset threshold, the target image is the first image.
[0008] Optionally, determining the similarity between the second image corresponding to the current frame and the second image corresponding to the previous frame includes: determining the difference in pixel values between corresponding positions in the second image corresponding to the current frame and the second image corresponding to the previous frame; and performing weighted processing on the difference in pixel values at each position to determine the similarity.
[0009] Optionally, weighted processing is performed on the difference in pixel values at each position, including: moving the second image corresponding to the current frame and the second image corresponding to the previous frame to the target moving position according to a specified step size with the user's gaze point as the center, so as to obtain the first pixel area corresponding to the target moving position of the second image corresponding to the current frame, and the second pixel area corresponding to the target moving position of the second image corresponding to the previous frame; summing the differences in pixel values at corresponding positions in the first pixel area and the second pixel area to obtain a summed result corresponding to the target moving position; performing weighted processing according to the summed results corresponding to each target moving position; and determining the similarity based on the weighted processing result.
[0010] Optionally, weighted processing is performed based on the summation results corresponding to each target moving position, including: determining the weight of the summation results corresponding to each target moving position in the weighted processing based on the distance between each target moving position and the user's gaze point; and using the weight to perform weighted processing on the summation results corresponding to each target moving position.
[0011] Optionally, before determining the first image, the method further includes: if there is an area in the user gaze area that is not covered by the original image, performing coordinate translation on the user gaze area so that the user gaze area falls within the range of the original image.
[0012] Optionally, transmitting the target image according to the similarity includes: if the similarity is less than a preset threshold, transmitting the first image corresponding to the current frame and the second image corresponding to the current frame.
[0013] Optionally, the method further includes: obtaining the coordinates of the user's gaze point; and expanding a specified range in all directions with the coordinates of the user's gaze point as the center to determine the user's gaze area.
[0014] Optionally, the method further includes: using the second image corresponding to the current frame to update the cached second image corresponding to the previous frame.
[0015] According to a second aspect of the present disclosure, an image processing method is provided, including: receiving a target image of a current frame, the target image including a first image, the target image being determined based on the similarity between a second image corresponding to the current frame and a second image corresponding to a previous frame, the first image being determined from an original image of the current frame based on a user's gaze area, the second image being determined based on the original image of the current frame, and the resolution of the second image being lower than that of the first image; when the similarity is greater than or equal to a preset threshold, the target image is the first image; and performing image fusion processing on the received first image corresponding to the current frame and the second image corresponding to the previous frame to obtain an image to be displayed.
[0016] Optionally, the target image includes a first image corresponding to the current frame and a second image corresponding to the current frame, and the method further includes: performing image fusion processing on the first image corresponding to the current frame and the second image corresponding to the current frame to obtain an image to be displayed.
[0017] Optionally, the second image to be fused is restored to its original size to obtain a corresponding third image, so that the third image has the same size as the corresponding original image; the second image to be fused is the second image corresponding to the current frame or the second image corresponding to the previous frame; the first image and the third image are fused so that the first image covers the corresponding position of the third image to obtain a corresponding fourth image; the fusion boundary in the fourth image is smoothed to obtain the image to be displayed.
[0018] According to a third aspect of the present disclosure, an image processing device is provided, including an image determination module, a similarity determination module and a transmission module, wherein: the image determination module is configured to determine a corresponding first image from an original image of a current frame based on a user's gaze area; and based on the original image of the current frame, determine a second image corresponding to the current frame, where the resolution of the second image is lower than that of the first image; the similarity determination module is configured to determine the similarity between the second image corresponding to the current frame and the second image corresponding to the previous frame; and the transmission module is configured to transmit a target image based on the similarity; wherein, when the similarity is greater than or equal to a preset threshold, the target image is the first image.
[0019] According to a fourth aspect of the present disclosure, an image processing device is provided, comprising a receiving unit and a processing unit, wherein: the receiving unit is configured to receive a target image of a current frame, the target image comprising a first image, the target image being determined based on the similarity between a second image corresponding to the current frame and a second image corresponding to a previous frame, the first image being determined from an original image of the current frame based on a user's gaze area, the second image being determined based on the original image of the current frame, and a resolution of the second image being lower than that of the first image; when the similarity is greater than or equal to a preset threshold, the target image is the first image; the processing unit is configured to perform image fusion processing on the received first image corresponding to the current frame and the second image corresponding to the previous frame to obtain an image to be displayed.
[0020] According to a fifth aspect of the present disclosure, a display device is provided, comprising: an image processor and a display, wherein: the image processor is configured to execute the method as described in any one of the above embodiments; and the display is configured to display an image to be displayed.
[0021] According to a sixth aspect of the present disclosure, a virtual reality display system is provided, comprising the display device as described in the above embodiment.
[0022] According to a seventh aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the image processing method in the above embodiment is implemented.
[0023] According to an eighth aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the image processing method as in the above-mentioned embodiment.
[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.
[0026] FIG1 schematically shows one of the flowcharts of an image processing method according to an embodiment of the present disclosure.
[0027] FIG2 schematically shows a flow chart of a coordinate translation process of a first image according to an embodiment of the present disclosure.
[0028] FIG3( a ) schematically shows a schematic diagram of the positions of an original image and a first image of a current frame according to an embodiment of the present disclosure.
[0029] FIG3( b ) schematically shows a first image of a current frame according to an embodiment of the present disclosure.
[0030] FIG3( c ) schematically shows a second image of the current frame according to an embodiment of the present disclosure.
[0031] FIG4 schematically shows a schematic diagram of pixel areas corresponding to target movement positions divided based on user gaze points according to an embodiment of the present disclosure.
[0032] FIG5 schematically shows a second flowchart of an image processing method according to another embodiment of the present disclosure.
[0033] FIG6 schematically shows a flowchart of image fusion processing according to an embodiment of the present disclosure.
[0034] FIG7 schematically shows a schematic diagram of a cubic filtering process according to an embodiment of the present disclosure.
[0035] FIG8 schematically shows a third flowchart of an image processing method according to an embodiment of the present disclosure.
[0036] FIG9 schematically shows one of the block diagrams of an image processing apparatus according to an embodiment of the present disclosure.
[0037] FIG10 schematically shows a second block diagram of an image processing apparatus according to an embodiment of the present disclosure.
[0038] FIG11 schematically shows a block diagram of a display device according to an embodiment of the present disclosure.
[0039] FIG12 schematically shows a module diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0040] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.
[0041] In addition, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present disclosure.
[0042] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0043] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.
[0044] In one embodiment of the present disclosure, an image processing method is proposed. FIG1 shows a flow chart of the image processing method. As shown in FIG1 , the image processing method includes at least the following steps:
[0045] Step S110 , determining a corresponding first image from the original image of the current frame according to the user's gaze area; and determining a second image corresponding to the current frame based on the original image of the current frame, where the resolution of the second image is lower than that of the first image.
[0046] Step S120 , determining the similarity between the second image corresponding to the current frame and the second image corresponding to the previous frame.
[0047] Step S130 : transmitting the target image according to the similarity; wherein, when the similarity is greater than or equal to a preset threshold, the target image is the first image.
[0048] The image processing method of the disclosed embodiment, on the one hand, determines the first image of the current frame based on the user's gaze area and transmits the first image, which can ensure the image clarity of the user's area of interest and the information integrity of the user's area of interest, thereby ensuring the user experience. On the other hand, based on the original image of the current frame, the second image of the current frame can be determined, and the resolution of the second image is lower than the first image. When the similarity between the second image of the current frame and the second image of the previous frame is greater than a preset threshold, only the first image of the current frame is transmitted. When the similarity between the two adjacent frames is high, the data transmission volume of the current frame can be reduced, the network bandwidth pressure can be reduced, the real-time data transmission can be guaranteed, the phenomenon of freezing and high latency can be avoided, and the user experience can be improved.
[0049] In order to make the technical solution of the present disclosure clearer, each step of the image processing method is described below.
[0050] In step S110 , a corresponding first image is determined from the original image of the current frame according to the user's gaze area; and a second image corresponding to the current frame is determined based on the original image of the current frame.
[0051] In this exemplary embodiment, the original image of the current frame can be subjected to image compression processing to reduce the resolution of the original image and obtain a second image corresponding to the current frame. For example, the original image can be subjected to a mean pooling operation to compress the original image. The compression parameters of the image can be set according to actual conditions, such as compressing it to 1 / 4 of the original image through 2*2 mean pooling. The second image can be a low-resolution original image, and the resolution of the second image is lower than that of the first image. As shown in Figure 3, Figure 3(a) is the original image of the current frame, and the box in Figure 3(a) indicates the position of the first image; Figure 3(b) is the first image corresponding to the current frame, and Figure 3(c) is the second image corresponding to the current frame. This example can reduce the bandwidth pressure of network transmission by reducing the resolution of the transmitted image.
[0052] In this exemplary embodiment, the original image refers to a high-resolution image to be transmitted, such as a 4k*4k image. For example, the user's gaze area in the current frame of the original image can be used as the first image. Alternatively, the user's gaze area can be appropriately enlarged or reduced to serve as the first image, although this example is not limited thereto.
[0053] In some embodiments, the coordinates of the user's gaze point may be obtained first; and then a designated range may be expanded around the coordinates of the user's gaze point to determine the user's gaze area.
[0054] In this exemplary embodiment, eye tracking technology can be used to determine the coordinates of the user's gaze point. Eye tracking technology is based on the physiological structure of the eyeball and can include, for example, any one of the sclera-iris edge method, pupil tracking method, and pupil-corneal reflection method.
[0055] For example, the pupil-corneal reflection method is described below: On the device side, an eye tracker located around the pupil of a device (such as a VR device or mobile phone) emits infrared light to illuminate the user's eyes. An infrared camera captures the infrared light reflected from the cornea and retina. Due to the physical properties of the eyeball, if the relative position of the light source and the head remains unchanged, the spot of light reflected from the cornea does not move, while the light reflected from the retina is emitted from the pupil. Therefore, the direction of the light reflected from the retina indicates the orientation of the pupil. The angle between the light reflected from the cornea and the pupil can be used to determine the direction of eye movement, and thus the gaze point in the camera coordinate system. The gaze point in the camera coordinate system is converted to the image coordinate system to obtain the coordinates of the gaze point in the image. The device side sends these coordinates to the image processing side (such as an image rendering server) at a certain frequency (e.g., several hundred hertz). The image processing side uses the gaze point coordinates as the center point and expands a specified range in all directions. For example, as shown in Figure 2, a 960*960 area centered on point O is defined as the user's gaze area (indicated by the dashed box in the figure).
[0056] In some embodiments, if there is an area in the user's gaze area that is not covered by the original image, the coordinates of the user's gaze area are translated so that the user's gaze area falls within the range of the original image.
[0057] In this exemplary embodiment, when the user's gaze point is at the edge of the image, there may be areas of the user's gaze area that are not covered by the original image, as shown in the left figure of Figure 2. To address this situation, the entire user's gaze area can be coordinate-translated to move it within the original image. For example, the coordinates of the boundary line of the uncovered portion of the user's gaze area can be subtracted from the coordinates of the corresponding boundary line in the original image to determine the required distance of movement in each axis. As shown in the left figure of Figure 2, the Y-axis movement Δy can be determined by subtracting the ordinates of the boundaries A1B1 and A2B2; the X-axis movement Δx can be determined by subtracting the abscissas of the boundaries A1C1 and A2C2 and taking their absolute values. The user's gaze area A1B1C1D1 is then shifted by the above-mentioned amount so that A1B1C1D1 falls within A2B2C2D2, as shown in the right figure of Figure 2. In some embodiments, when the user's gaze area is completely covered by the original image, the first image can be saved as a texture. This example uses coordinate translation to ensure the validity of the user's gaze area when the user's gaze point is at the edge of the image.
[0058] In some embodiments, the second image corresponding to the current frame may be used to update the cached second image corresponding to the previous frame.
[0059] In this exemplary embodiment, when the second image corresponding to the current frame is obtained, the second image can be cached. To save computing resources, the second image corresponding to the current frame can be used to update the second image corresponding to the previous frame.
[0060] In step S120 , the similarity between the second image corresponding to the current frame and the second image corresponding to the previous frame is determined.
[0061] In this exemplary embodiment, image similarity can be determined in various ways. For example, similarity can be measured by distance (such as absolute distance, Euclidean distance, Manhattan distance, etc.), or feature extraction can be performed first and then similarity coefficients can be calculated (such as cosine similarity, various similarity coefficients, etc.).
[0062] For example, the difference in pixel values between corresponding positions in the second image corresponding to the current frame and the second image corresponding to the previous frame may be determined first; and then the difference in pixel values at each position may be weighted to determine the similarity.
[0063] In this exemplary embodiment, the pixel values at corresponding positions in the second image corresponding to adjacent frames can be firstly differentiated (i.e., the absolute distance can be calculated), and then the differences in pixel values at different positions can be weighted. The weight of the weighted processing can be determined based on the distance between the position corresponding to the pixel difference and the user's gaze point. For example, a larger weight is set for a position closer to the user's gaze point, and a smaller weight is set for a position farther away. The weight can satisfy a certain relationship with the distance between the pixel position and the user's gaze point, such as a cubic Bezier curve, a sigmoid curve, or a relationship formed by adjusting the above curve relationship, which is not limited in this example. In some embodiments, normalization can be performed before weighting.
[0064] Exemplarily, the second image corresponding to the current frame and the second image corresponding to the previous frame can be moved to the target moving position in a specified step size with the user's gaze point as the center, so as to obtain the first pixel area corresponding to the target moving position of the second image corresponding to the current frame and the second pixel area corresponding to the target moving position of the second image corresponding to the previous frame; the difference between the pixel values at corresponding positions in the first pixel area and the second pixel area is summed to obtain the summation result corresponding to the target moving position; weighted processing is performed according to the summation results corresponding to each target moving position; and the similarity is determined based on the weighted processing result.
[0065] In this exemplary embodiment, the specified step size can be set based on actual conditions and can be set to one or several pixels, etc., but this example does not limit this. During each movement, the user's gaze point is used as the center and moves radially in a specified step size to the target movement position. The pixel area corresponding to the target movement position is selected. Pixels with the same or similar distance from the user's gaze point can be divided into the same pixel area. That is, each movement corresponds to a target movement position, and each target movement position corresponds to a pixel area. Each movement is performed separately in the second image corresponding to the current frame and the second image corresponding to the previous frame, respectively, to obtain the first pixel area corresponding to the target movement position in the second image corresponding to the current frame and the second pixel area corresponding to the target movement position in the second image corresponding to the previous frame. The two second images undergo the same movement operation. As shown in Figure 4, the movement process is explained using the second image corresponding to the current frame as an example. Point O is the user's gaze point in the second image corresponding to the current frame. Moving around point O, the first movement obtains pixel area 1, the second movement obtains pixel area 2, the third movement obtains pixel area 3, and the fourth movement obtains pixel area 4.
[0066] After obtaining the first pixel area and the second pixel area corresponding to the two second images, the difference between the pixel values of the corresponding positions of the first pixel area and the second pixel area can be summed, that is, the pixel difference of the two pixel areas obtained for each target moving position is summed, and then the summation results corresponding to each target moving position are weighted.
[0067] For example, the weighted processing result corresponding to each target moving position can be calculated by the following formula:
[0068] Wherein, sum represents the weighted processing result of the two second images, W j represents the weight of the jth target position, j represents the target position's index, n represents the total number of pixels in the pixel region corresponding to the target position, i represents the pixel index in the pixel region corresponding to the target position, and m represents the total number of target positions. δr, δg, and δb represent the differences between the R, G, and B pixel values at the corresponding positions in the two second images, respectively, and |·| represents the absolute value. This formula calculates the absolute difference of the R, G, and B pixel values for each pixel position in the region, sums them, and averages them. The result is then normalized by dividing by 255 to obtain a weighted value ranging from 0 to 1. This process continues until the target position reaches the image boundary, at which point the sum value is obtained.
[0069] In this example, the similarity is determined based on the weighted processing result, so that the weighted processing result is inversely proportional to the similarity. For example, the inverse of the weighted processing result can be used as the similarity, or a mapping relationship can be established between the weighted processing result and the similarity, which is not limited in this example.
[0070] Exemplarily, the weighted processing may be: determining the weight of the summation result corresponding to each target moving position in the weighted processing according to the distance between each target moving position and the user's gaze point; and performing weighted processing on the summation result corresponding to each target moving position using the weight.
[0071] In this exemplary embodiment, the weight of each distance corresponding to the target movement position can be used to perform a weighted summation on the sum of the pixel differences between the first pixel region and the second pixel region corresponding to the target movement position. Alternatively, the summation results can be normalized, averaged, or otherwise processed before performing the weighted summation. This example is not limited to this. For example, the weight of each pixel region can be determined based on a modified sigmoid curve, as shown in the following formula:
[0072] Where s represents the distance from the target moving position j to the user's gaze point.
[0073] For example, Figure 4 shows a schematic diagram of the weights of each target's moving position satisfying a cubic Bezier curve. In Figure 4, the user's gaze point in the second image corresponding to the current frame is point O, which is divided into four pixel regions. Pixel region 1 has a weight of 0.45, pixel region 2 has a weight of 0.35, pixel region 3 has a weight of 0.15, and pixel region 4 has a weight of 0.05.
[0074] In the above-mentioned embodiment of the present disclosure, the weight of the weighted processing can be determined according to the distance between the target moving position and the user's gaze point, thereby introducing the user attention factor in the similarity calculation process, setting a larger weight for the area with high user attention (the position close to the gaze point), and setting a smaller weight for the area with low user attention (the position far from the gaze point). While considering the difference in pixel values of the two second images, the influence of user attention is also taken into account, which can reduce the impact on the user experience brought by using the second image corresponding to the previous frame to replace the second image corresponding to the current frame to process the image, thereby ensuring user experience.
[0075] In step S130 , the target image is transmitted according to the similarity.
[0076] In this exemplary embodiment, if the similarity is greater than or equal to a preset threshold, indicating that the difference between the two adjacent image frames is small, the target image is the first image. Specifically, only the first image of the current frame is transmitted, and the area of the current frame other than the area corresponding to the first image is not transmitted. This reduces the data transmission volume of the current frame and bandwidth pressure, while ensuring the clarity of the first image corresponding to the user's gaze area and user experience. The preset threshold can be set based on the complexity of the actual image scene. For example, when the scene is simple, the preset threshold can be set higher. When the scene is complex, the difference between the two image frames may be large, so the preset threshold can be set lower.
[0077] In some embodiments, before transmitting the data, the data to be transmitted, i.e., the target image, may be subjected to transmission encoding processing to facilitate network transmission. After receiving the data (the encoded target image), the receiving end may decode the received data (the encoded target image) to recover the data.
[0078] In some embodiments, if the similarity is less than a preset threshold, the first image corresponding to the current frame and the second image corresponding to the current frame are transmitted.
[0079] In this exemplary embodiment, if the similarity is less than a preset threshold, indicating a significant difference between two adjacent frames (the current frame and the previous frame), the first image corresponding to the current frame and the corresponding second image are transmitted. In this example, when the difference between adjacent frames is significant, a low-resolution image of the original image and a high-resolution image corresponding to the user's gaze area are transmitted. While ensuring clarity in the user's area of interest, the transmission of the low-resolution image reduces the bandwidth pressure associated with transmitting full HD images, meets real-time requirements, and improves the user experience.
[0080] Referring to FIG5 , the present disclosure further provides an image processing method, which is applied to a device and may include the following steps:
[0081] Step S510, receiving a target image of the current frame, where the target image includes a first image, and the target image is determined based on the similarity between the second image corresponding to the current frame and the second image corresponding to the previous frame. The first image is determined from the original image of the current frame based on the user's gaze area, and the second image is determined based on the original image of the current frame, and the resolution of the second image is lower than that of the first image. When the similarity is greater than or equal to a preset threshold, the target image is the first image.
[0082] Step S520 , performing image fusion processing on the received first image corresponding to the current frame and the received second image corresponding to the previous frame to obtain an image to be displayed.
[0083] In this exemplary embodiment, when the received target image is a first image, image fusion processing can be performed using the first image corresponding to the current frame and the second image corresponding to the previous frame to fuse the high-resolution first image into the low-resolution second image, forming an image with high resolution in the user's gaze area and low resolution in other areas.
[0084] In some embodiments, when the target image includes a first image corresponding to the current frame and a second image corresponding to the current frame, image fusion processing is performed on the first image corresponding to the current frame and the second image corresponding to the current frame to obtain an image to be displayed.
[0085] Exemplarily, the image fusion processing may include the following steps: first, restoring the size of the second image to be fused to obtain a corresponding third image, so that the third image has the same size as the corresponding original image; fusing the first image and the corresponding third image so that the first image covers the corresponding position of the third image to obtain a corresponding fourth image; and smoothing the fusion boundary in the fourth image to obtain the image to be displayed.
[0086] In this exemplary embodiment, the second image to be fused can be the second image corresponding to the current frame or the second image corresponding to the previous frame. The second image to be fused can be determined based on the target of the image fusion process. In this example, upsampling can be used to restore the size of the first image, determine the position of the first image within the third image, overlay the first image at the corresponding position of the third image, and then smooth the fusion boundary between the first and third images to obtain the image to be displayed. For example, the images during this image fusion process are shown in Figure 6.
[0087] In this exemplary embodiment, smoothing refers to smoothing the boundary between high-resolution and low-resolution images. Smoothing can include one or more filtering processes, such as mean filtering or Gaussian filtering. The types of filtering processes in different rounds can be the same or different, and this example does not limit this. As shown in Figure 7, three Gaussian filters can be applied to the fused boundary, with the three Gaussian filters being applied successively from low resolution to high resolution. The order of filtering in Figure 7 is 1, 2, and 3.
[0088] Referring to FIG8 , an image processing method in another embodiment provided by the present disclosure may include the following steps:
[0089] In step S801, the device determines the coordinates of the current user gaze point through eye tracking.
[0090] Step S802: The device sends the coordinates of the user's gaze point to the server.
[0091] Step S803: The server determines the user's gaze area based on the coordinates of the user's gaze point.
[0092] Step S804: The server determines whether the user's gaze area is completely covered by the original image. If so, the process goes to S805; otherwise, the process goes to S806.
[0093] Step S805: The server performs coordinate translation on the user's gaze area so that it falls within the original image range, and then goes to S805.
[0094] Step S806: The server determines a first image corresponding to the original image of the current frame according to the user's gaze area.
[0095] Step S807: The server performs image compression processing on the original image of the current frame to obtain a second image corresponding to the current frame.
[0096] Step S808: The server determines the similarity between the second image corresponding to the current frame and the second image corresponding to the previous frame.
[0097] In step S809, the server determines whether the similarity is greater than or equal to a preset threshold. If so, the process goes to S810; otherwise, the process goes to S811.
[0098] In step S810 , the server determines that the target image is the first image corresponding to the current frame, and then goes to S812 .
[0099] Step S811: The server determines that the target image is the first image corresponding to the current frame and the second image corresponding to the current frame, and then goes to S811.
[0100] Step S812: The server transmits the target image to the device.
[0101] In step S813, the device determines whether the target image includes the second image. If so, the process goes to S814; otherwise, the process goes to S815.
[0102] In step S814 , the device performs image fusion processing on the first image corresponding to the current frame and the second image corresponding to the current frame to obtain an image to be displayed, and then goes to S816 .
[0103] In step S815 , the device performs image fusion processing on the first image corresponding to the current frame and the second image corresponding to the previous frame to obtain an image to be displayed, and then goes to S816 .
[0104] Step S816: The device displays the image to be displayed.
[0105] In the above embodiments, the image fusion processing process can be executed in the processor on the device side or in the display on the device side, which is not limited in this example.
[0106] The image processing method provided by the present disclosure, on the one hand, determines the first image of the current frame based on the user's gaze area and transmits the first image, which can ensure image clarity and information integrity in the user's area of interest, thereby ensuring a better user experience. On the other hand, based on the original image of the current frame, a second image of the current frame can be determined, where the resolution of the second image is lower than the first image. When the similarity between the second image of the current frame and the second image of the previous frame is greater than a preset threshold, only the first image of the current frame is transmitted. This can reduce the amount of data transmitted for the current frame when the similarity between two adjacent frames is high, thereby alleviating network bandwidth pressure, ensuring real-time data transmission, avoiding lag and high latency, and improving the user experience.
[0107] The following describes an embodiment of the device disclosed herein, which can be used to perform the image processing method disclosed herein. For details not disclosed in the embodiment of the device disclosed herein, please refer to the embodiment of the image processing method disclosed herein.
[0108] FIG9 schematically shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure.
[0109] 9 , an image processing apparatus 900 according to an embodiment of the present disclosure includes: an image determination module 910, a similarity determination module 920, and a transmission module 930. Specifically:
[0110] The image determination module 910 is configured to determine the corresponding first image from the original image of the current frame based on the user's gaze area; and based on the original image of the current frame, determine the second image corresponding to the current frame, where the resolution of the second image is lower than that of the first image; the similarity determination module 920 is configured to determine the similarity between the second image corresponding to the current frame and the second image corresponding to the previous frame; the transmission module 930 is configured to transmit the target image based on the similarity; wherein, when the similarity is greater than or equal to a preset threshold, the target image is the first image.
[0111] In some embodiments of the present disclosure, based on the aforementioned scheme, the similarity determination module 920 includes: a pixel difference determination submodule and a weighted processing submodule, wherein: the pixel difference determination submodule is configured to determine the difference in pixel values between the corresponding positions in the second image corresponding to the current frame and the second image corresponding to the previous frame; the weighted processing submodule is configured to perform weighted processing on the difference in pixel values of each position to determine the similarity.
[0112] In some embodiments of the present disclosure, based on the aforementioned scheme, the weighted processing submodule is further configured to: move the second image corresponding to the current frame and the second image corresponding to the previous frame to the target moving position according to a specified step size with the user's gaze point as the center, so as to respectively obtain the first pixel area corresponding to the target moving position of the second image corresponding to the current frame, and the second pixel area corresponding to the target moving position of the second image corresponding to the previous frame; sum the differences between the pixel values at corresponding positions in the first pixel area and the second pixel area to obtain the summation result corresponding to the target moving position; perform weighted processing according to the summation results corresponding to each target moving position; and determine the similarity based on the weighted processing result.
[0113] In some embodiments of the present disclosure, based on the aforementioned scheme, the weighted processing submodule is further configured to: determine the weight of the summation result corresponding to each target moving position in the weighted processing according to the distance between each target moving position and the user's gaze point; and use the weight to perform weighted processing on the summation result corresponding to each target moving position.
[0114] In some embodiments of the present disclosure, based on the aforementioned scheme, the device 900 also includes a translation module, and the translation module is configured to: before determining the first image, if there is an area of the user's gaze area that is not covered by the original image, coordinate translation of the user's gaze area is performed so that the user's gaze area falls within the range of the original image.
[0115] In some embodiments of the present disclosure, based on the aforementioned solution, the transmission module 930 is further configured to: if the similarity is less than a preset threshold, transmit the first image corresponding to the current frame and the second image corresponding to the current frame.
[0116] In some embodiments of the present disclosure, based on the aforementioned solution, the device 900 further includes: an acquisition module, which is configured to: acquire the coordinates of the user's gaze point; and expand a specified range around the coordinates of the user's gaze point to determine the user's gaze area.
[0117] In some embodiments of the present disclosure, based on the aforementioned solution, the apparatus 900 further includes: a cache update module, the cache update module being configured to update the cached second image corresponding to the previous frame using the second image corresponding to the current frame.
[0118] Figure 10 is a block diagram of an image processing device according to some other embodiments of the present disclosure. As shown in Figure 10, according to an image processing device 1000 of one embodiment of the present disclosure, the image processing device 1000 includes: a receiving unit 1010 and a processing unit 1020, wherein: the receiving unit 1010 is configured to receive a target image of a current frame, the target image includes a first image, the target image is determined according to the similarity between the second image corresponding to the current frame and the second image corresponding to the previous frame, the first image is determined from the original image of the current frame according to the user's gaze area, the second image is determined based on the original image of the current frame, and the resolution of the second image is lower than the first image; when the similarity is greater than or equal to a preset threshold, the target image is the first image; the processing unit 1020 is configured to perform image fusion processing on the received first image corresponding to the current frame and the second image corresponding to the previous frame to obtain an image to be displayed.
[0119] In some embodiments of the present disclosure, based on the aforementioned scheme, the target image includes a first image corresponding to the current frame and a second image corresponding to the current frame, and the processing unit 1020 is further configured to: perform image fusion processing on the first image corresponding to the current frame and the second image corresponding to the current frame to obtain the image to be displayed.
[0120] In some embodiments of the present disclosure, based on the aforementioned scheme, the processing unit 1020 is further configured to: restore the size of the second image to be fused to obtain a corresponding third image, so that the third image is the same size as the corresponding original image; the second image to be fused is the second image corresponding to the current frame or the second image corresponding to the previous frame; fuse the first image and the corresponding third image so that the first image covers the corresponding position of the third image to obtain the corresponding fourth image; smooth the fusion boundary in the fourth image to obtain the image to be displayed.
[0121] The specific details of each of the above-mentioned image processing devices have been described in detail in the corresponding image processing methods, and therefore will not be repeated here.
[0122] An embodiment of the present disclosure also provides a display device, comprising the image processing device of any of the aforementioned embodiments. Figure 11 shows a block diagram of a display device according to some embodiments of the present disclosure. Referring to Figure 11, a display device provided by the present disclosure comprises: an image processor 1110 and a display 1120, wherein: the image processor 1110 is configured to execute the image processing method of any of the aforementioned embodiments. The display 1120 is configured to display the image to be displayed. In some embodiments, the display may include at least one of a liquid crystal display and an OLED (Organic Light-Emitting Diode) display. In some embodiments, the display device may be: any product or component with a display function, such as a mobile phone, a tablet computer, a television, a laptop computer, a digital photo frame, a navigator, or the like.
[0123] The present disclosure also provides a virtual reality display system, including the display device of any of the aforementioned embodiments. Utilizing the display device of the present disclosure can ensure the real-time performance of a large-scale VR system.
[0124] It should be noted that although several modules or units of the device for execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0125] In this exemplary embodiment, an electronic device capable of implementing the above method is also provided.
[0126] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."
[0127] The electronic device 1200 according to this embodiment of the present invention is described below with reference to Figure 12. The electronic device 1200 shown in Figure 12 is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.
[0128] As shown in FIG12 , electronic device 1200 is implemented as a general-purpose computing device. Components of electronic device 1200 may include, but are not limited to, the aforementioned at least one processing unit 1210, the aforementioned at least one storage unit 1220, a bus 1230 connecting various system components (including storage unit 1220 and processing unit 1210), and a display unit 1240.
[0129] The storage unit stores program code, which can be executed by the processing unit 1210, so that the processing unit 1210 performs the steps of various exemplary embodiments of the present invention described in the image processing method described above in this specification. For example, the processing unit 1210 can perform step S110 as shown in Figure 1: determining a corresponding first image from the original image of the current frame based on the user's gaze area; and based on the original image of the current frame, determining a second image corresponding to the current frame, where the resolution of the second image is lower than that of the first image; step S120: determining the similarity between the second image corresponding to the current frame and the second image corresponding to the previous frame; step S130: transmitting a target image based on the similarity; wherein, if the similarity is greater than or equal to a preset threshold, the target image is the first image.
[0130] The storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 12201 and / or a cache memory unit 12202 , and may further include a read-only memory unit (ROM) 12203 .
[0131] The storage unit 1220 may also include a program / utility 12204 having a set (at least one) of program modules 12205, such program modules 12205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0132] The bus 1230 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0133] Electronic device 1200 can also communicate with one or more external devices (e.g., a keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a viewer to interact with electronic device 1200, and / or any device that enables electronic device 1200 to communicate with one or more other computing devices (e.g., a router, modem, etc.). Such communication can occur via input / output (I / O) interface 1250. Furthermore, electronic device 1200 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via network adapter 1260. As shown, network adapter 1260 communicates with other modules of electronic device 1200 via bus 1230. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with electronic device 1200, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0134] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
[0135] In this exemplary embodiment, a computer-readable storage medium is also provided, storing a program product capable of implementing the methods described above. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.
[0136] According to an embodiment of the present invention, a program product for implementing the above-mentioned method can be a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, and the program can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0137] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0138] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0139] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0140] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and the like, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0141] Furthermore, the figures above are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.
[0142] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.
[0143] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. An image processing method, characterized in that: include: Determining a corresponding first image from an initial image of a current frame according to the user gaze area; and determining, based on the original image of the current frame, a second image corresponding to the current frame, wherein the resolution of the second image is lower than that of the first image; Determine the similarity between the second image corresponding to the current frame and the second image corresponding to the previous frame; According to the similarity, a target image is transmitted; wherein, when the similarity is greater than or equal to a preset threshold, the target image is the first image.
2. The method according to claim 1, characterized in that Determining the similarity between the second image corresponding to the current frame and the second image corresponding to the previous frame includes: Determine the difference in pixel values between corresponding positions in the second image corresponding to the current frame and the second image corresponding to the previous frame; The difference between the pixel values at each position is weighted to determine the similarity.
3. The method according to claim 2, characterized in that The difference between the pixel values at each position is weighted, including: The second image corresponding to the current frame and the second image corresponding to the previous frame are respectively moved to the target moving position according to the specified step length with the user's gaze point as the center, so as to respectively obtain the first pixel area corresponding to the target moving position of the second image corresponding to the current frame and the second pixel area corresponding to the target moving position of the second image corresponding to the previous frame; Summing the differences between the pixel values at corresponding positions in the first pixel area and the second pixel area to obtain a summation result corresponding to the target moving position; Perform weighted processing according to the summation results corresponding to the moving positions of each target; The similarity is determined according to the weighted processing result.
4. The method according to claim 3, characterized in that Weighted processing is performed according to the summation results corresponding to the moving positions of each target, including: Determining the weight of the summation result corresponding to each target moving position in the weighted processing according to the distance between each target moving position and the user's gaze point; The weights are used to perform weighted processing on the summation results corresponding to the moving positions of the targets.
5. The method according to claim 1, characterized in that Before determining the first image, the method further includes: If there is an area in the user gaze area that is not covered by the original image, coordinate translation is performed on the user gaze area so that the user gaze area falls within the range of the original image.
6. The method according to claim 1, characterized in that The transmitting the target image according to the similarity comprises: If the similarity is less than a preset threshold, the first image corresponding to the current frame and the second image corresponding to the current frame are transmitted.
7. The method according to claim 1, characterized in that The method further comprises: Get the coordinates of the user's gaze point; The user gaze area is determined by expanding a designated range around the coordinates of the user gaze point as the center.
8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: The second image corresponding to the previous frame in the cache is updated using the second image corresponding to the current frame.
9. An image processing method, characterized in that: include: Receive a target image of a current frame, the target image comprising a first image, the target image being determined according to a similarity between a second image corresponding to the current frame and a second image corresponding to a previous frame, the first image being determined from an original image of the current frame according to a user gaze area, the second image being determined based on the original image of the current frame, and having a lower resolution than the first image; When the similarity is greater than or equal to a preset threshold, the target image is the first image; An image fusion process is performed on the received first image corresponding to the current frame and the received second image corresponding to the previous frame to obtain an image to be displayed.
10. The method according to claim 9, characterized in that The target image includes a first image corresponding to the current frame and a second image corresponding to the current frame, and the method further includes: An image fusion process is performed on the first image corresponding to the current frame and the second image corresponding to the current frame to obtain the image to be displayed.
11. The method according to claim 9 or 10, characterized in that: The image fusion processing includes: Resizing the second image to be fused to obtain a corresponding third image, so that the third image has the same size as the corresponding original image; the second image to be fused is the second image corresponding to the current frame or the second image corresponding to the previous frame; fusing the first image and the third image so that the first image covers a corresponding position of the third image to obtain a corresponding fourth image; The fusion boundary in the fourth image is smoothed to obtain the image to be displayed.
12. An image processing device, characterized in that: include: An image determination module is configured to determine a corresponding first image from an original image of a current frame according to a user gaze area; and determining, based on the original image of the current frame, a second image corresponding to the current frame, wherein the resolution of the second image is lower than that of the first image; a similarity determination module, configured to determine a similarity between a second image corresponding to a current frame and a second image corresponding to a previous frame; The transmission module is configured to transmit a target image according to the similarity; wherein, when the similarity is greater than or equal to a preset threshold, the target image is the first image.
13. An image processing device, characterized in that: include: The receiving unit is configured to receive a target image of a current frame, wherein the target image includes a first image, wherein the target image is determined based on a similarity between a second image corresponding to the current frame and a second image corresponding to a previous frame, wherein the first image is determined from an original image of the current frame based on a user gaze area, and the second image is determined based on a current frame. The target image is determined by the original image of the previous frame, and the resolution of the second image is lower than that of the first image; when the similarity is greater than or equal to a preset threshold, the target image is the first image; The processing unit is configured to perform image fusion processing on the received first image corresponding to the current frame and the second image corresponding to the previous frame to obtain an image to be displayed.
14. A display device, comprising: An image processor configured to execute the method according to any one of claims 9 to 11; The display is configured to display the image to be displayed.
15. A virtual reality display system, comprising: The display device as claimed in claim 14.
16. An electronic device comprising: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method according to any one of claims 1 to 11.
17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.
Citation Information
Patent Citations
Video playback method, server, virtual reality device, and panoramic virtual reality playback system
CN107040794A
Video playing method and system and equipment
CN108810636A
Image compression method and device and related equipment
CN112153388A
Display method, device and system, storage medium and displayer
CN112819967A
Image processing method and device, display device and virtual reality display system
CN116600128A
