Processing method and device for generating depth map of three-dimensional VR (virtual reality) view

Abstract

The invention relates to an advanced processing method and device for stereoscopic three-dimensional VR (Virtual Reality). Disclosed is an image processing method, comprising: receiving a 2D image, receiving a depth map corresponding to the 2D image, the depth map comprising an original depth value of a corresponding pixel of pixels of the 2D image; receiving focus area information about the eye; calculating the distance between the pixel in the 2D image or the depth map and the center of the focus area; reconfiguring a corresponding depth map of the 2D image by compressing or magnifying original depth values of pixels of at least one distance range according to a specified depth value compression ratio or magnification based on the calculated distance between the pixels and the center of the focus region to attenuate or enhance 3D perception of the corresponding distance range region; and outputting the reconfigured depth map for generating a 3D view based on the 2D image.

Description

Technical Field

[0001] The present disclosure generally relates to the field of virtual reality (VR) display, and in particular, to a processing method and apparatus for generating a depth map of a stereoscopic three-dimensional (3D) VR. Background Art

[0002] Virtual reality (VR) systems often use head-mounted displays to present stereoscopic VR content to users. Stereoscopic VR creates a 3D sense of depth in the 360-degree stereoscopic video displayed on the head-mounted display, which is closer to a realistic and immersive viewing experience. The emergence of three-dimensional display technology has made virtual reality a reality. However, although viewers can experience the visual enjoyment brought by stereoscopic videos in an immersive and interactive way, if viewers continue to watch stereoscopic videos for a certain period of time, they will cause discomfort symptoms such as eye fatigue and even headaches. This phenomenon has greatly hindered the development and application of stereoscopic 3D display technology. In addition, some viewers will have a strong sense of dizziness, fatigue, vertigo, nausea, etc. after experiencing stereoscopic VR content. These are symptoms of VR motion sickness. VR motion sickness not only prevents users from immersing themselves in the virtual world for a long time, but also makes the user's repetition rate extremely low. Therefore, how to reduce the visual fatigue caused by watching stereoscopic 3D videos and improve VR motion sickness while achieving a 3D effect that is closer to the real human eye has become one of the urgent problems to be solved in the process of popularizing virtual reality, and has also become a hot topic of research and industry in recent years.

[0003] It is recognized that the visual acuity of the human eye generally decreases with distance from the foveal (central visual field). In order to simulate the decrease in acuity from the eye socket to the periphery in the visual system, as experienced by humans, VR head-mounted displays that integrate eye tracking technology often use a graphics computing technology called foveated rendering. This technology reduces visual fatigue and reduces computational complexity by reducing the image quality of the image outside the focal / fixation point, thereby effectively reducing the GPU's calculations and thus reducing power consumption.

[0004] For example, the technical document "Foveated Brightness Control Technology for VR Applications, Won-Been Jeong etc." discloses a foveated brightness control technology for VR applications to overcome the hardware limitations of VR devices. The document is based on the theory that people can perceive high-resolution images in the focal area, that is, the gaze point, so the resolution outside this area can be reduced, which helps to display the image that the eye actually sees while reducing image processing and thus reducing power consumption. The document proposes to determine the foveated area according to the gaze point of the eye, and gradually reduce the brightness of the non-foveated area according to the gaze point of the eye.

[0005] The technical document "Modelling Foveated Depth-of-field Blur for Improving DepthPerception in Virtual Reality, Razeen Hussain etc." discloses a depth blur model for the foveal area to improve depth perception in VR. This document develops a system for VR devices to mimic the visual experience of the real world. First, out-of-focus blur is applied based on focus and depth of field to reduce visual fatigue. Second, foveal rendering is applied based on multiple areas of focus. This document uses a combination of the two to produce a more natural scene, and helps reduce the computational load of VR devices by rendering the content around the foveal area into a smaller frame buffer resolution.

[0006] U.S. patent application US20220148538A1 discloses a method and system for depth-based concave rendering in a display system. The display system may be an augmented reality display system configured to provide virtual content on multiple depth planes using different wavefront divergences. Some embodiments include determining a gaze point of a user's eyes. Position information associated with a first virtual object is obtained, and the first virtual object is presented to the user through a display device. A resolution modification parameter of the first virtual object is obtained. Based on the position information and the resolution modification parameter of the first virtual object, a specific resolution for rendering the first virtual object is identified. The specific resolution is based on a resolution distribution specified for a corresponding distance from the gaze point. The first virtual object rendered at the identified resolution is presented to the user through the display system.

[0007] US patent application US20220197381A1 discloses using eye tracking to adjust a region of interest (ROI) to compress a transmitted image. The patent application adjusts the compression of one or more transmitted images so that when transmitting data of an image portion outside one or more regions of interest, fewer bits are required than the number of bits of the image portion within one or more regions of interest.

[0008] PCT patent application WO2019 / 183430 A1 discloses a device for displaying images of regions with different bit depths. The device may include a display device, which includes an overall display receiving bit depth allocation data, and configuring the overall display based on the bit depth allocation data to display image data at different bit depths in various display regions of the overall display. This can cause the display device to consume a lower proportion of image data to drive the overall display, and maintain a higher image quality in the display region of the overall display that is configured to display image data at a higher bit depth.

[0009] The information provided in this "Background" section is for the purpose of generally presenting the background of the present disclosure. The description in this section relates to work prior to the filing of the presently named inventors and does not constitute prior art and is neither explicitly nor implicitly admitted to be prior art to the present disclosure. Summary of the invention

[0010] Different from the concept in the prior art of reducing visual fatigue by reducing image quality in non-central areas, the inventors of the present application found that when rendering a 3D view, adjusting the corresponding depth map with respect to the fovea (central visual field) of the eye in the 2D image can enhance the true 3D visual experience of the VR visual system and reduce 3D VR motion sickness caused by excessive depth perception, while reducing stereo artifacts and distortion.

[0011] Based on such a finding, an image processing method and apparatus for enhancing 3D visual experience are proposed in an embodiment of the present disclosure.

[0012] According to a first aspect of the present disclosure, there is provided an image processing method, comprising receiving a 2D image, receiving a depth map corresponding to the 2D image, wherein the depth map comprises original depth values ​​of pixels corresponding to the 2D image pixels; receiving focus area information about an eye; calculating the distance between the 2D image pixels and the center of the focus area; reconfiguring the depth map of the 2D image by compressing or amplifying the original depth values ​​of the 2D image pixels in at least one distance range according to a specified depth value compression rate or amplification rate based on the calculated distance between the pixels and the center of the focus area, so as to weaken or enhance the 3D perception of the corresponding distance range area; and outputting the reconfigured depth map for generating a 3D view based on the 2D image.

[0013] According to a second aspect of the present disclosure, a head-mounted display device is provided, comprising: a tracking unit for detecting a focus area of ​​a user's eyes; a depth map processing unit, comprising a memory storing computer instructions, which, when executed by one or more processors, executes the aforementioned method to output a reconfigured depth map; a 3D engine for generating a 3D image based on the reconfigured depth map and the 2D image; and a display unit for displaying the generated 3D stereoscopic image.

[0014] According to a third aspect of the present disclosure, there is provided an image processing apparatus, comprising: a data receiving module, configured to receive a 2D image, a depth map corresponding to the 2D image, and information about a focus area of ​​a user's eyes, wherein the depth map comprises original depth values ​​representing pixels of the 2D image; a focus area analysis unit, configured to calculate a distance between a pixel on the 2D image and a center of the focus area; and a depth scaling module, reconfiguring the depth map of the 2D image by compressing or amplifying original depth values ​​of pixels of the 2D image in at least one distance range according to a specified depth value compression rate or amplification rate based on the calculated distance between the pixel and the center of the focus area, so as to weaken or enhance 3D perception of a corresponding distance range area.

[0015] These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 Schematically shows a schematic diagram of generating a 3D view by performing 3D rendering according to a 2D color image 101 and a corresponding depth map 102; Figure 2A flow chart showing an exemplary embodiment of adjusting 3D perception by configuring a depth map corresponding to a 2D color image according to the present invention is shown; Figure 3 A schematic diagram schematically shows a method of calculating the distance of a pixel in the depth map 102 based on the focus / fixation point determined based on the information of the focus / fixation point area; Figure 4A A linear compression curve graph 400A showing linear compression of original depth values ​​according to one embodiment of the present disclosure; Figure 4B shows a compressed depth map obtained by linearly compressing the original depth value according to the linear compression curve graph 400A; Figure 4C A nonlinear compression curve diagram 400C showing nonlinear compression of original depth values ​​according to an embodiment of the present disclosure; Figure 4D shows a compressed depth map obtained by nonlinearly compressing the original depth value according to the nonlinear compression curve graph 400C; Figure 5A and Figure 5B A preferred embodiment of the present disclosure is schematically shown; Figure 6 A schematic diagram showing a method of configuring a pixel depth weight based on a distance x between a pixel in a depth map and a focus / gaze point according to an embodiment of the present disclosure is shown; Fig. 7A An embodiment of parameter setting of the conversion function f(x) is shown; Figure 7B Another embodiment of parameter setting of the conversion function f(x) is shown; Figure 8 A schematic diagram of generating a fused depth map according to an embodiment of the present disclosure is shown; Fig. 9 A schematic diagram schematically shows that a compressed depth map reduces artifacts in a 3D view; Fig.10 An illustrative embodiment of a depth map with two-level viewing areas is shown; Fig.11A A schematic diagram of generating a fused depth map according to another embodiment of the present disclosure is shown; Fig. 11B A schematic diagram of generating a fused depth map according to another embodiment of the present disclosure is shown; Fig.12 A schematic diagram showing the reduction of distortion in a 3D view by a compressed depth map; Fig.13 A block diagram schematically shows an image device according to an embodiment of the present disclosure; and Fig.14A block diagram of a computing device according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0017] Before describing the embodiments of the present disclosure in detail, some related concepts are first explained.

[0018] Focus / Fixation point: refers to the point in the visual field that the two eyes fixate on under normal vision. For each eye, it is the point that directly stimulates the fovea centralis or fovea. The fovea, or fovea, is a small depression in the neurosensory retina where the visual acuity of the human eye is the highest. The fovea itself is the central part of the macula and is responsible for central vision. In head-mounted displays, information about the focus / fixation point is usually obtained using eye gaze information obtained by an eye tracking unit.

[0019] The foveated area, also referred to herein as the central visual field area, is the area around the focus / attention point in the image. In the present invention, the depth value compression rate of an object in the image can vary among pixels of the entire image according to one or more "focus / attention points", with pixels in the focus / attention point having the lowest depth compression rate, corresponding to the center of the fovea of ​​the eye's retina.

[0020] The unfoveated area around the fovea is also referred to as the non-central visual field in this article: the image has a lower resolution and a higher depth compression rate than the image in the fovea.

[0021] As mentioned above, the visual acuity of the human eye generally decreases with the distance from the fovea (central visual field). The inventors of the present application realize that the 3D perception of the human eye also decreases with the distance from the fovea (central visual field). In addition, different users may have different acceptance of 3D perception. In view of this, the inventors of the present application propose that when a 3D view is generated by a 3D rendering engine, a depth map corresponding to a 2D image is configured for the acquired focus / fixation point information so that personalized 3D perception can be customized to obtain a more realistic and comfortable 3D visual experience, while reducing 3D VR motion sickness caused by excessive depth perception. In addition, this technology can reduce stereoscopic artifacts and distortion while generating a more realistic and comfortable 3D view. This is specifically described in the following embodiments.

[0022] Figure 1 The schematic diagram of generating a 3D view by 3D rendering based on a 2D color image 101 and a corresponding depth map 102 is shown schematically. Figure 1The 2D RGB color image 101 and the corresponding depth map 102 shown in the figure are input into the 3D rendering engine 103 to generate a 3D view. From the perspective of the image dimension, the depth map 102 is also a 2D image, which is the same size as the corresponding RGB color image 101, except that the grayscale value corresponding to each pixel p(x, y) represents the relative distance from a point on the object in the image to, for example, the viewpoint. Each pixel p(x, y) of the depth map pixel has an 8-bit grayscale value, whose value is between [0-255] or [black, white], where 255 (white) represents the object closest to the viewpoint, and 0 (black) represents the object farthest from the viewpoint. In this application, the grayscale value corresponding to each pixel p(x, y) is called a depth value.

[0023] The 3D rendering engine 103 then generates a 3D stereoscopic view including a pair of views 104 and 105 based on the 2D RGB color image 101 and the corresponding depth map 102. The pair of views 104 and 105 are respectively presented as a right eye view and a left eye view of a VR display so that the audience has a 3D sense of depth and thus obtains a more realistic and immersive viewing experience. Those skilled in the art will appreciate that the 3D view is not limited to two views, the right eye view and the left eye view, but may also be multiple views.

[0024] Image acquisition devices can use depth sensors to determine the depth or relative distance of objects in the scene to obtain corresponding depth maps while capturing 2D color images. Typically, these depth sensors rely on the capture of reflections of known spatially modulated or temporally modulated light projected by the device on the object. For example, this modulated light can be infrared (IR) light generated by an infrared light source. Then, a corresponding infrared-specific sensor captures the reflected infrared light and processes the captured infrared image for depth sensing or other visual telemetry purposes. In the absence of a depth camera to obtain a depth map, a monocular camera can also be used to indirectly calculate the depth map through related algorithms. A well-known method is Structure from Motion, such as moving (usually requiring large movements) a monocular camera to obtain multiple pictures from different perspectives to indirectly obtain a depth map. In addition to obtaining a corresponding depth map while capturing a 2D color image through hardware methods, a partial or full depth map of a 2D color image can also be generated manually or with the help of an AI algorithm. For example, in the case of a sparse image captured only by a depth sensor, a dense pixel-level depth map can be predicted by algorithms such as machine learning. In addition, when only a single 2D image is available, a three-dimensional mesh can be directly obtained from the single 2D image through deep learning to obtain a corresponding depth map.

[0025] From the above-mentioned depth map acquisition method, it can be seen that the depth map obtained by the depth sensor or the depth camera or the depth map generated manually or with the aid of the AI ​​algorithm is intended to reflect the real physical distance between the object and the camera in the shooting scene. Therefore, the 3D view generated based on such depth map rendering will give the audience a consistent 3D perception in the entire field of view. However, the inventors of the present application realize that the 3D perception of the human eye for objects in the field of view also decreases with the increase of the distance from the fovea (central field of view). In order to simulate the decline of 3D perception from the eye socket to the periphery in the visual system to obtain a more realistic and comfortable 3D experience, the inventors of the present application propose that when rendering the 3D view, the 3D perception is adjusted by adjusting the depth value of the pixel depth map obtained in the traditional manner for the fovea (central field of view), so that the image with more depth perception is presented to the center of the user's field of view, while the image with reduced depth perception is presented to the peripheral area of ​​the user's central field of view. By using this uneven visual processing, the user can obtain a more realistic and comfortable 3D experience, and reduce visual fatigue and VR motion sickness caused by excessive 3D perception. In addition, considering that different users may have different acceptance of 3D perception, the present application also provides a solution that allows further personalization of 3D perception in order to obtain a more realistic and comfortable 3D visual experience.

[0026] Figure 2 A flow chart of an exemplary embodiment of adjusting 3D perception by configuring a depth map corresponding to a 2D image according to the present invention is shown. In step S201, a 2D image of a scene is received. The 2D image of the scene can be obtained by, for example, an RGB camera, and in step S202, a depth map corresponding to the 2D image is received, the depth map including the original depth values ​​of the pixels corresponding to the pixels of the 2D image, such as the depth values ​​corresponding to the pixels of the 2D image obtained by the depth sensor or depth camera of the image acquisition device while acquiring the 2D image. In order to present an image with more depth perception to the user's central field of vision and present an image with reduced depth perception to the peripheral area of ​​the user's central field of vision, it is necessary to first determine the center of the user's field of vision. This can be done in step S203 by deriving information about the focus / fixation point area from, for example, eye gaze information captured by an eye tracking system. Then, it is necessary to calculate the distance between the 2D image pixel and the center of the focus / fixation point area in step S204. It is understandable that the distance between the corresponding pixel of the depth map and the center of the focus / fixation point area can also be calculated.

[0027] Figure 3 Schematic diagram 300 is shown for calculating the distance of a pixel in the depth map 102 based on the focus / gaze point determined by the information of the focus / gaze point area. Figure 3As shown, taking the focus / gaze point 303 (x0, y0) as the origin of the coordinates, the distance of any pixel with coordinates (x, y) on the depth map can be calculated according to the following formula: sqrt{(x-x0)^2 + (y-y0)^2}. Figure 3 The pixel distance at the focus / gaze point 303 in the image is 0, the pixel distance at the object 301 is 0.38, and the pixel distance at the object 302 is 0.73. In this embodiment, the distance of the pixel farthest from the focus / gaze point 303 (x0, y0) in the image is 1.

[0028] Then, in step S205, the depth map of the 2D image is reconfigured by compressing or amplifying the original depth values ​​of the 2D image pixels in at least one distance range according to the desired 3D perception effect based on the calculated distance between the pixel and the focus / fixation point. In one embodiment, the desired 3D effect can be achieved by compressing or amplifying the original depth values ​​of the pixels at a specified depth value compression rate or amplification rate. In one embodiment, the depth value compression or amplification of the pixel is achieved by converting the original depth value of the pixel in the numerical range of [0-255] to a depth value in the compressed or amplified numerical range proportional to the desired depth value compression rate or amplification rate.

[0029] For ease of explanation, the specific process of compressing the depth value of a pixel is described below by taking compressing the original depth value of the pixel to reduce the 3D perception as an example. Those skilled in the art can amplify the original depth value of the pixel to enhance the 3D perception based on a similar working principle.

[0030] Figure 4A A linear compression curve graph 400A for linearly compressing the original depth value according to an embodiment of the present invention is schematically shown. The horizontal axis of the compression curve graph 400A is the original depth value of the pixel, and the vertical axis is the compressed depth value of the pixel. The original value range of the original depth value of the pixel is [0-255]. If the original value range is [0-255] and the median value 128 is used as the center to uniformly compress inward, the minimum value L and the maximum value H of the value range of the compressed depth value and the compressed depth value CompDepth can be calculated according to the following equation: L = int(0 + 255*compression_ratio / 2) H = int(255 - 255*compression_ratio / 2) CompDepth = int((OrgDepth / 255) * (H - L) + L) When the compression ratio compression_ratio is specified as 0%, 50%, 70% and 90%, the ranges of the compressed depth values ​​are [0-255], [63-191], [89-165], [114-140], respectively. After linear compression at compression ratios of 0%, 50%, 70% and 90%, the relationship between the original depth value and the corresponding compressed depth value is shown as lines 401, 402, 403, 404. For example, if the original depth value OrgDepth is 100, when the compression ratio is 90%, i.e., 0.9, the minimum value L and the maximum value H of the compressed value range, as well as the depth value obtained after linear compression are calculated as follows: L = int(0 + 255*0.9 / 2) = 114 H = int(255 - 255*0.9 / 2) = 140 CompDepth = int((100 / 255)*(140-114)+114) = 124 It is understandable that the numerical interval after compression according to the depth compression rate is reduced in proportion to the original numerical interval. For example, when the specified depth compression rate is 90%, the length of the compressed numerical interval [114-140] is 10% of the original numerical interval [1-255], that is, (100%-90%). Based on the above description, a person skilled in the art can infer that all equivalent alternatives or obvious variants can achieve the same effect of depth value compression or amplification, and all such equivalent alternatives or obvious variants belong to the disclosure scope of this application.

[0031] The oblique lines 401-404 in the linear compression curve graph 400A correspond to the correspondence between the original depth value and the compressed depth value when the original depth value is linearly compressed at compression rates of 0%, 50%, 70% and 90%, respectively. The dotted lines in the compression curve graph 400 indicate the value range after compression at different compression rates. If the depth value of the pixel in the depth map has a wide dynamic range, for example, it changes in the entire value range [0-255], then applying such a linear compression of the depth value can achieve better results.

[0032] Figure 4BThe depth map in FIG. 4 shows that after all pixels of the original depth map 102 with a depth value range of [0-255] are compressed at a compression rate of 50% according to the linear compression curve graph 400A, a compressed depth map 410 with a depth value range of [63-191] is obtained, after compression at a compression rate of 75%, a compressed depth map 420 with a depth value range of [89-165] is obtained, and after compression at a compression rate of 90%, a compressed depth map 430 with a depth value range of [114-140] is obtained. It can be seen from the compressed depth maps 410, 420, and 430 that as the depth compression rate increases, the grayscale contrast of the depth map pixels gradually weakens. It can be understood that when a 3D view is rendered based on a depth map, the higher the grayscale value, that is, the whiter the place is, the more convex it is perceived by the audience, and the darker the place is, the more concave it is perceived by the audience. Since the grayscale contrast of the depth map is weakened after depth compression, the stereoscopic 3D perception of the final rendered 3D view is correspondingly weakened.

[0033] In another embodiment, the original depth value may also be compressed nonlinearly. Figure 4C A nonlinear compression curve graph 400C is shown for nonlinear compression of the original depth value according to an embodiment of the present invention. For nonlinear compression, the numerical range of the depth value after compression is calculated in the same way as during linear compression. If the original numerical range is [0-255] and the median 128 is used as the center for uniform compression, when the compression ratio compression_ratio is specified as 0%, 50%, 70% and 90%, the numerical range of the compressed depth value is [0-255], [63-191], [89-165], [114-140] respectively. After nonlinear compression with compression ratios of 0%, 50%, 70% and 90%, the relationship between the original depth value and the corresponding compressed depth value is shown in curves 401, 402, 403 and 404. For the same original depth value, the compressed depth values ​​obtained after nonlinear compression and linear compression may be different. For example, when the specified compression ratio is 90%, the depth value CompDepth after nonlinear compression of the original depth value 100 can be calculated as: CompDepth = int(1.0 / (1 + exp(-0.05 * (OrgDepth - 128)))) * (140 -114) + 114) = 119 This is different from the compressed depth value 124 obtained after the previous linear compression.

[0034] If the dynamic range of the depth value of the pixel in the depth map is narrow, for example, the depth value of the pixel changes only in a part of the entire value range [0-255], such as the range of [50-200], then applying nonlinear compression of the depth value can achieve better results. It can be seen from the nonlinear compression curve 400C that in this embodiment, the depth value close to the median retains more changes after compression.

[0035] Figure 4D The depth maps 440, 450, 460 and 470 in FIG. 4 respectively show different depth maps obtained after nonlinear compression at different compression rates. Specifically, Figure 4D The diagram shows a depth map 440 obtained by compressing all pixels of the original depth map 102 with a depth value range of [0-255] at a compression rate of 0% (no compression, only nonlinearization) according to the nonlinear compression curve graph 400C, a compressed depth map 450 with depth values ​​in the range of [63-191] obtained by compressing at a compression rate of 50%, a compressed depth map 460 with depth values ​​in the range of [89-165] obtained by compressing at a compression rate of 75%, and a compressed depth map 470 with depth values ​​in the range of [114-140] obtained by compressing at a compression rate of 90%.

[0036] In another embodiment, when compressing the depth value, the compression may not be performed inwardly with the midpoint of the original depth value range as the center, so the median of the compressed depth value range may be different from the median of the original depth value range. It may also be compressed to other intervals of the original value range [0-255] according to the actual needs of adjusting depth perception.

[0037] The above describes an embodiment of compressing all pixels on the depth map at a specified compression rate. However, as mentioned above, the 3D perception of the human eye for objects in the field of view also decreases with the distance from the fovea (central field of view). In order to simulate the decline of 3D perception from the eye socket to the periphery in the visual system to obtain a more realistic and comfortable 3D experience, depth compression can be performed on the foveal (central field of view) and the peripheral sub-areas of the central field of view.

[0038] Figure 5A and Figure 5B A preferred embodiment of the present invention is shown, in which, in the fovea / central visual field within a certain distance range from the focus / fixation point 501 (x0, y0), for example Figure 5A and Figure 5BThe fovea 502 shown in the figure retains the original depth value of the pixel, and only the depth value of the pixel in the non-central field of view outside the fovea 502 is compressed at a specified compression rate to reduce the depth perception of the peripheral area of ​​the user's central field of view, that is, the non-central field of view. In this way, images with more depth perception can be presented to the center of the user's field of view, while images with reduced depth perception can be presented to the peripheral area of ​​the central field of view. By using this uneven visual processing, users can obtain a more realistic and comfortable 3D experience, reducing visual fatigue and VR motion sickness caused by excessive 3D perception.

[0039] Will be like Figure 1 The original depth map 102 shown in FIG. 1 can be obtained by performing the above adjustment. Figure 5B In the depth map 504 shown, the pixels in the central visual area within the fovea 502 retain the original depth values, thereby retaining the original 3D perception of the objects in the central visual area, while the depth values ​​of the pixels in the non-central visual area outside the fovea 502 are compressed at a compression rate of 90%, thereby weakening the human eye's 3D perception of the objects in the non-central visual area of ​​the scene.

[0040] In such Figure 5B In the embodiment shown in the depth map 504, the pixels of the central field of view 502 retain the original depth values ​​to retain the original 3D perception, while the depth values ​​of the pixels outside the central field of view are compressed, thereby weakening the 3D perception of the human eye in the non-central field of view scene. As mentioned above, there are different requirements for 3D stereo effects in different scenarios, and different users have different acceptance of 3D perception. In order to meet different scenarios and user needs to provide a more comfortable and realistic stereoscopic 3D experience, the present application proposes a solution to provide a personalized 3D experience by flexibly configuring depth weights for the pixels of the depth map to allow users to adjust the depth perception effect of the 3D view according to their needs. Specifically, the solution proposed in the present application allows the user to adjust the size of the central field of view by selecting parameters and to adjust the smoothness of the change between different depth compression rates.

[0041] Figure 6 A schematic diagram of configuring pixel depth weights based on the distance x between a pixel of a depth map and a focus / gaze point according to an embodiment of the present disclosure is shown. Graph 601 shows, in one embodiment, configuring a depth weight map of a pixel of a depth map 603 according to the following conversion function f(x) relative to the distance of the pixel. (1) Where: x = -x if x>0, parameter c1 is used to adjust the smoothness of the change in depth weight from the central visual field with high depth perception to the non-central visual field with low depth perception, and parameter c2 is used to adjust the range of the central visual field with high depth perception.

[0043] exist Figure 6 In the illustrated embodiment, the depth weight of the pixel near the focus / attention point with the highest depth perception is set to 1, and the depth weight of the pixel far from the focus / attention point with the weakest depth perception is set to 0. According to the above conversion function f(x), the depth weight of the pixel at any distance x from the focus / attention point in the image can be determined.

[0044] As shown in the distance graph 603, the distance of the pixel at the focus / fixation point 6001 is 0, the distance at the object 6002 is 0.38, and the distance at the object 6003 is 0.73. As shown in Table 602, according to the conversion function f(x), for the distances of the pixels 0, 0.38, 0.73, when the parameter c1 is set to 40 and c2 is set to -0.35, the depth weights at 6001, 6002 and 6003 are 1, 0.20 and 0 respectively. The weight graph 601 shows the weight at any distance x from the focus / fixation point obtained according to the conversion function f(x). The depth graph 604 shows the depth graph after the above configuration. It can be seen from the depth map 604 that the middle white part 6005 is a pixel whose weight is kept as 1. This part can be used as the central field of view, where the pixels can use the original depth value to retain the original 3D perception, while the peripheral black part is used as the non-central field of view, where the depth value of the pixel is compressed at a specified compression rate, thereby weakening the 3D perception of the human eye in the scene of the non-central field of view. The range of pixels whose weight is kept as 1 can be adjusted by the conversion function c2. The larger the absolute value of c2, the larger the range of pixels whose weight is kept as 1, and the larger the central field of view, that is, the larger the range of the white part in the weight map 604. The gray value between the white part 6005 with a weight of 1 and the black part 6006 with a weight of 0 is gradually reduced, and the smoothness of the change of this part can be adjusted by the parameter c1. The larger the value of the parameter c1, the more rapid the change. In the weight map, it is shown that the change from weight 0 to weight 1 is more rapid. On the contrary, the smaller the value of the parameter c1, the smoother the change from weight 0 to weight 1.

[0045] Fig. 7A The weight graphs 701, 702, 703 and the corresponding conversion function curves are shown when c2 is kept at -0.35 and c1 is set to 20, 40, 200 respectively. Fig. 7A It can be seen that as the c1 value increases, the change from weight 0 to weight 1 becomes more rapid.

[0046] Figure 7B The weight graphs 704, 705, 706 and the corresponding conversion function curves are shown when c1 is kept at 40 and c2 is set to -0.15, -0.35, -0.55 respectively. Figure 7BAs can be seen in , when c1 remains unchanged, the larger the absolute value of c2 is, the larger the range of the white portion whose weight is kept as 1 in the weight graphs 704, 705, and 706 is, that is, the larger the range of the central visual field with high depth perception is.

[0047] It is understandable that other conversion functions can be used to set the depth weight according to actual needs.

[0048] Figure 8 A schematic diagram of generating a fused depth map using a weight map according to a specified compression rate according to an embodiment of the present disclosure is shown. In this embodiment, the depth value of the central field of view remains unchanged, that is, the compression rate is 0%, and the highest compression rate of the non-central field of view outside the central field of view is specified as 90%. When the parameter c1 is set to 40 and c2 is set to -0.35, the depth weight map 803 is obtained according to the conversion function f(x). In the weight map 803, the depth weight of the whitest part of the pixels in the central field of view 805 is 1, and the depth weight of the blackest part of the pixels in the non-central field of view 806 is 0, and the depth value of this part of the pixels is compressed at a maximum compression rate of 90%. The weight of the gradient part of the pixels between the whitest and the blackest gradually changes from 1 to 0, that is, the compression rate of the gradient part of the pixels increases from 0% to 90%.

[0049] The compressed depth value of a pixel in the fused depth map 904 may be calculated according to the following equation: Fusion depth value = original depth value * depth weight + compressed depth value * (1-depth weight).

[0050] Fig. 9 A schematic diagram showing that artifacts are reduced by compressed depth maps. Another technical effect of depth compression according to an embodiment of the present disclosure is to reduce artifacts in a stereoscopic view of 3D rendering.

[0051] Figure 901 shows the views of the left and right eyes obtained after 3D rendering using the original depth map without any compression. The pixel depth values in the original depth map may vary within a relatively large range, and correspondingly, they appear very protruding or very recessed in the stereoscopic view. In this case, due to the strong 3D effect, artifacts are likely to appear in the 3D-rendered stereogram. For example, as can be seen from Figure 901, the nose of the portrait in the circle 903 in the left-eye view is very prominent, while the nose of the portrait in the circle 904 in the right-eye view can hardly be seen. When users see such a stereoscopic view, they will have a mismatched and ghostly visual experience, which is also one of the reasons for causing VR motion sickness. Performing 3D rendering based on the depth map processed by the depth compression method proposed in this application can reduce the artifacts in the 3D-rendered stereoscopic view. Figure 902 is the views of the left and right eyes obtained after 3D rendering using the depth map compressed at a compression rate of 90%. From the portrait in the circle 905 in the left-eye view, it can be seen that the nose is not as prominent as before, and the nose also appears in the portrait in the circle 906 in the right-eye view. In this way, the artifacts appearing in the 3D-rendered stereoscopic view can be reduced, and the distortion of the stereoscopic view is improved. It can be understood that for the purpose of reducing artifacts and improving distortion, the depth values of the pixels in the central visual field area can also be compressed. If the situation of artifacts or distortion is relatively serious, a larger compression rate can be used for compression.

[0052] The depth weight can gradually change from the pixel area with a depth weight of 1 in the central visual field area to the pixel area with a depth weight of 0 in the non-central visual field area in any mode. In one embodiment, multiple levels of target depth weights can be specified between the pixels with a depth weight of 1 and the pixels with a depth weight of 0. The pixel areas assigned with their respective level target weights can be regarded as one-level visual field areas. The assignment of each level of target depth weight and how to gradually change between the levels of target depth weights can be configured by a function f M (x) for each level of visual field area. On the weight map, the multiple levels of target depth weights can be presented as multiple concentric rings, and the gray values of the rings gradually decrease as the distance of the pixels from the focus / attention point increases.

[0053] In one embodiment, assuming that the amplitude difference between the changes of the levels of target depth weights is average, when there are N levels of visual field areas, the following conversion functions f 1 (x) to f M (x) are used to configure the weights for the pixels in the 1st to the Mth level of visual field areas, where M < N, the depth weight amplitude AmplitudeM = (1 / N), and the offset OffsetM from the focus / attention point with coordinates (0, 0) is (1 - M / N).

[0054] For f 1(x), parameter c1 is used to adjust the smoothness of the change from pixels with the first level specified weight to pixels with the second level specified weight, and parameter c2 is used to adjust the range of pixels with the first level specified weight. Similarly, for f 2 (x), parameter c3 is used to adjust the smoothness of the change from pixels with the second-level specified weight to pixels with the third-level specified weight, and parameter c4 is used to adjust the range of pixels with the second-level specified weight. M (x), parameter c(2M-1) is used to adjust the smoothness of the change from pixels with the Mth level specified weight to pixels with the M+1th level specified weight, and parameter c2M is used to adjust the range of pixels with the Mth level specified weight.

[0055] Fig.10 An exemplary embodiment with two levels of viewing areas is shown. In this embodiment, the weight change range is evenly distributed between the first level viewing area and the second level viewing area. In this embodiment, the first level depth weight is 1, the second level depth weight is 0.5, and the change range of each level is 0.5. 1 (x) Assign depth weights to the first-level visual field. The first-level visual field includes the area from the focus / gaze point with a depth weight of 1 pixel to the pixel with a second-level depth weight of 0.5. 2 (x) assigning a depth weight to the second-level field of view, the second-level field of view including an area from pixels with a depth weight of 0.5 to pixels with a depth weight of 0. In this embodiment, the offset relative to the focus / gaze point is also evenly distributed between the two-level field of view. In this embodiment, the distance of the farthest pixel in the image is 1, the offset of the first-level field of view is 0.5, and the offset of the second-level field of view is 0. Chart 1001 shows the conversion function with multiple fields of view of this embodiment, and weight map 1002 shows the weight map of the pixels of the multiple fields of view of this embodiment.

[0056] Fig.11A FIG. 4 is a schematic diagram showing a fused depth map generated according to another embodiment of the present disclosure. Fig.10 The depth weight map 1002 with two-level viewing areas of the embodiment is used to generate a fused depth map 1104 with two-level viewing areas. In this embodiment, the depth values ​​of the pixels of the white first-level viewing area 1105 in the depth weight map 1002 are kept as the depth values ​​of the original depth map 1101, that is, the lowest compression rate is 0% and the depth weight is 1; the black non-central viewing area 1106 has the highest specified compression rate of 90%, the depth weight is 0, and the linear compression depth map shown in the depth map 1102 is used. The highest depth weight in the pixels of the gray second-level viewing area 1107 between the white first-level viewing area 1105 and the black non-central viewing area 1106 is 0.5. According to the aforementioned conversion function f1 (x) Configuring the weights of pixels from the first-level viewing area 1105 with a depth weight of 1 to the second-level viewing area 1107 with a depth weight of 0.5, and 2 (x) configures the weights of the pixels from the second-level viewing area 1107 with a depth weight of 0.5 to the pixels in the non-central viewing area 1106 with a depth weight of 0. 1 (x) and f 2 The parameters c1 of (x) are set to 100, c2 to 0.35, c3 to 100, and c4 to -0.2.

[0057] Fig. 11B FIG. 4 is a schematic diagram showing a fused depth map generated according to another embodiment of the present disclosure. Fig.11A The same depth weight map 11002 with two-level viewing areas as in the embodiment of the present invention is used to generate a fused depth map 1110 with two-level viewing areas. In this embodiment, the depth values ​​of the pixels in the white first-level viewing area 1105 are kept as the depth values ​​of the original depth map 1101, that is, the lowest compression rate is 0% and the depth weight is 1; the black non-central viewing area 1106 has the highest specified compression rate of 90% and the depth weight is 0, which is the same as the original depth map 1101. Fig.11A The difference of the embodiment is that nonlinear compression is used as shown in the depth map 1109. The highest depth weight of the pixels in the gray second-level field of view 1107 between the white first-level field of view 1105 and the black non-central field of view 1106 is 0.5. 1 (x) Configuring the weights of pixels from the first-level viewing area 1105 with a depth weight of 1 to the pixels in the second-level viewing area 1107 with a depth weight of 0.5, and 2 (x) configures the weights of the pixels from the second-level viewing area 1107 with a depth weight of 0.5 to the pixels in the non-central viewing area 1106 with a depth weight of 0. 1 (x) and f 2 The parameters c1 of (x) are set to 100, c2 to 0.35, c3 to 100, and c4 to -0.2.

[0058] Fig.12 A schematic diagram showing the reduction of distortion in a 3D view by a compressed depth map. Another technical effect of depth compression according to an embodiment of the present disclosure is to improve the distortion of a stereoscopic view.

[0059] Figure 1201 shows the left and right eye views obtained after 3D rendering using the original depth map without any compression. For example, it can be seen from Figure 1201 that the images of the two door handles of the building door in circles 1205 and 1206, which should have been vertical, have become distorted in the left and right eye views, with obvious distortion. Figure 1202 shows the 3D rendering using the original depth map without any compression. Fig. 11B The left and right eye views obtained after 3D rendering with the fused depth map 1110 in FIG. The fused depth map 1110 is a depth map obtained by specifying a 90% compression rate for nonlinear compression and fusion with a multi-level field of view depth weight map. From FIG. 1202 , it can be seen that the distortion of the image of the building handles in circles 1205 and 1206 in the left eye view and the right eye view is improved, and the distortion becomes less obvious.

[0060] Fig.13 The block diagram of an image processing device according to an embodiment of the present disclosure is schematically shown. The image processing device includes a data receiving module 1301, which is configured to receive a 2D color image, a depth map corresponding to the 2D color image, and information about the focus / gaze point of the user's eyes. The received depth map includes the original depth value of the corresponding pixel representing the 2D image pixel. The image processing device also includes a focus analysis module 1303, which is configured to calculate the distance between the pixel on the 2D image and the center of the focus area, and generate a depth weight map according to a conversion function based on the distance map. The image processing device also includes a depth compression module 1302, which compresses the original depth value of the 2D image pixel in at least one distance range according to a specified depth value compression rate based on the calculated distance between the pixel and the center of the focus area to generate a compressed depth value of the pixel. The image processing device also includes a processing module 1304, which performs depth fusion based on the specified depth value compression rate and the depth weight map to generate a compressed depth map. Finally, a 3D rendering engine generates multiple 3D stereoscopic views based on the compressed depth map and the 2D color image.

[0061] Although specific functions are discussed above with reference to specific modules, it should be noted that the functions of the various modules discussed herein can be divided into at least two modules, and / or at least some functions of at least two modules can be combined into a single module. In addition, the specific module discussed herein performing an action includes the specific module itself performing the action, or alternatively the specific module calling or otherwise accessing another component or module that performs the action (or performing the action in conjunction with the specific module). Therefore, the specific module that performs the action can include the specific module itself that performs the action and / or another module that the specific module that performs the action calls or otherwise accesses.

[0062] Above about Fig.13The various modules described may be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules may be implemented as computer program codes / instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules may be implemented as hardware logic / circuits. For example, in some embodiments, one or more of the data receiving module 1301, the depth compression module 1302, the focus analysis module 1303, the processing module 1304, and the 3D rendering engine 1305 may be implemented in an X86-based computing architecture, a graphics processing unit (GPU)-based computing architecture, or a system on chip (SoC). The SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform functions. The features of the technology described herein are carrier-independent, meaning that these technologies can be implemented on various computing platforms with various processors.

[0063] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer program. For example, an embodiment of the present disclosure provides a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing at least one step in the method embodiment of the present disclosure.

[0064] Fig.14 The block diagram of a computing device 1400 according to an embodiment of the present disclosure is schematically shown. The computing device 1400 can implement Fig.13 The image processing device includes one or more of a data receiving module 1301, a depth compression module 1302, a focus analysis module 1303, a processing module 1304 and a 3D rendering engine 1305.

[0065] Computing device 1400 may be a variety of different types of devices, such as a server computer, a device associated with a client (eg, a client device), a system on a chip, and / or any other suitable computing device or computing system.

[0066] Computing device 1400 may include at least one processor 1402, memory 1404, a communication interface 1406, a display device 1408, other input / output (I / O) devices 1410, and one or more mass storage devices 1412 that can communicate with each other, such as via a system bus 1414 or other appropriate connection.

[0067] The processor 1402 may be a single processing unit or at least two processing units, all of which may include a single or at least two computing units or at least two cores. The processor 1402 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. In addition to other capabilities, the processor 1402 may be configured to obtain and execute computer-readable instructions stored in the memory 1404, the mass storage device 1412, or other computer-readable media, such as program codes of an operating system 1416, program codes of an application program 1418, program codes of other programs 1420, etc., to implement the image recognition method 200 provided in the embodiment of the present disclosure.

[0068] The memory 1404 and the mass storage device 1412 are examples of computer storage media for storing instructions that are executed by the processor 1402 to implement the various functions described above. For example, the memory 1404 may generally include both volatile memory and non-volatile memory (e.g., RAM, ROM, etc.). In addition, the mass storage device 1412 may generally include a hard drive, a solid-state drive, a removable medium (including external and removable drives), a memory card, a flash memory, a floppy disk, an optical disk (e.g., a CD, a DVD), a storage array, a network storage, a storage area network, etc. The memory 1404 and the mass storage device 1412 may all be collectively referred to herein as memory or computer storage media, and may be a non-transitory medium capable of storing computer-readable, processor-executable program instructions as computer program codes that may be executed by the processor 1402 as a specific machine configured to implement the operations and functions described in the examples herein.

[0069] At least two program modules may be stored on the mass storage device 1412. These programs include an operating system 1416, one or more application programs 1418, other programs 1420, and program data 1422, and they may be loaded into the memory 1404 for execution. Examples of such applications or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the following parts / components / functions: data receiving module 1301, depth compression module 1302, focus analysis module 1303, processing module 1304, and 3D rendering engine 1305 and / or other embodiments described herein. In some embodiments, these program modules may be distributed in different physical locations to implement the corresponding functions.

[0070] Although in Fig.141400, but modules 1416, 1418, 1420, and 1422, or portions thereof, may be implemented using any form of computer-readable media accessible by the computing device 1400. As used herein, "computer-readable media" may include one or more types of computer-readable media, which may include, for example, computer storage media and / or communication media.

[0071] Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented by any method or technology for storage of information, such as computer readable instructions, data structures, program modules or other data. Computer storage media includes but is not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other non-transmission media that can be used to store information for access by a computing device.

[0072] In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism. Computer storage media as defined herein do not include computing device 1400 may also include one or more communication interfaces 1406 for exchanging data with other devices such as through a network, direct connection, etc. The communication interface 1406 may facilitate communication within a variety of network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. The communication interface 1406 may also provide communication with an external storage device (not shown) such as a storage array, network storage, storage area network, etc.

[0073] In some examples, a display device 1408 such as a monitor may be included for displaying information and images. Other I / O devices 1410 may be devices that receive various inputs from a user and provide various outputs to the user, and may include a touch input device, a gesture input device, a camera, a keyboard, a remote control, a mouse, a printer, an audio input / output device, and the like.

[0074] In a first embodiment, an image processing method includes receiving a 2D image, receiving a depth map corresponding to the 2D image, wherein the depth map includes original depth values ​​of corresponding pixels of the 2D image pixels; receiving information about a focus area of ​​an eye; calculating a distance between a pixel in the 2D image or the depth map and a center of the focus area; reconfiguring the depth map corresponding to the 2D image by compressing or amplifying original depth values ​​of pixels in at least one distance range according to a specified depth value compression rate or amplification rate based on the calculated distance between the pixel and the center of the focus area, so as to weaken or enhance 3D perception of a corresponding distance range area; and outputting the reconfigured depth map for generating a 3D view based on the 2D image.

[0075] A second embodiment may include the method according to the first embodiment, wherein the original depth value of the pixel has a first value range, and compressing or amplifying the original depth value of the pixel includes converting the original depth value of the pixel into a compressed depth value or an amplified depth value within a second value range according to a specified depth value compression ratio or amplification ratio.

[0076] A third embodiment may include a method according to the second embodiment, wherein, when the original depth value is magnified at a magnification greater than 1, the first value range is a subset of the second value range, and the second value range interval is proportionally magnified relative to the first value range according to the specified depth value magnification.

[0077] A fourth embodiment may include a method according to the second embodiment, wherein, when the depth value is compressed at a compression ratio less than 1, the second numerical range is a subset of the first numerical range, and the second numerical range interval is reduced proportionally relative to the first numerical range according to the specified depth value compression ratio.

[0078] A fifth embodiment may include a method according to the third embodiment or the fourth embodiment, wherein the median of the second numerical range is the same as the median of the first compressed or enlarged range, or the median of the second numerical range is different from the median of the first numerical range.

[0079] A sixth embodiment may include the method according to the fifth embodiment, wherein, for a given depth value compression or magnification ratio, the change in the conversion of the original depth value of the pixel into the compressed or magnified depth value within the second value range is non-linear.

[0080] A seventh embodiment may include the method according to the fifth embodiment, wherein, for a given depth value compression or magnification ratio, the change in the conversion of the original depth value of the pixel to the compressed or magnified depth value within the second value range is linear.

[0081] An eighth embodiment may include the method according to the sixth embodiment or the seventh embodiment, further comprising, for the 2D image pixels within the distance range, specifying a lowest level 1 target depth value compression rate and a highest level N target depth value compression rate for the pixels.

[0082] A ninth embodiment may include the method according to the eighth embodiment, further comprising: assigning depth weights to the 2D image pixels within the distance range to indicate the compression degree of pixels at a distance x from the center of the focus region according to a desired 3D perception degree, so that starting from the center of the focus region, the depth value compression rate of the 2D image pixels increases from a first level target depth value compression rate to an Nth level target depth value compression rate as the distance of the pixels increases; Here, a depth weight of 1 is assigned to pixels having a first-level target depth value compression rate, and a depth weight of 0 is assigned to pixels having an Nth-level target depth value compression rate.

[0083] A tenth embodiment may include the method according to the ninth embodiment, further comprising, specifying the 2D image pixels within the distance range to have target depth value compression rates from level 2 to level N-1, wherein any level M target depth value compression rate is assigned a corresponding depth weight value between 0 and 1, and wherein at least some of the 2D image pixels within the distance range have corresponding depth weights from level 1 to level N, respectively.

[0084] An eleventh implementation may include a method according to the tenth implementation, wherein the first-level target depth value compression rate is specified as 0, and the compressed depth value of the pixel within the distance range is calculated as: original depth value*depth weight+f(original depth value, Nth target depth value compression degree)*(1-depth weight).

[0085] A twelfth embodiment may include the method according to the eleventh embodiment, wherein according to the transfer function f M (x) to generate the depth weight distribution of the 2D image pixels from the weight value corresponding to any M-th level target depth value compression rate value to the weight value corresponding to the M+1-th level target depth value compression rate, and the conversion function f M (x) includes at least a first parameter c1 and a second parameter c2, wherein the first parameter c2 is used to adjust the range of the depth weight value corresponding to the Mth level target depth value compression rate of the 2D image pixels, and the second parameter c1 is used to adjust the change rate of the 2D image pixels from the depth weight value corresponding to the Mth level target depth value compression rate to the depth weight value corresponding to the M+1th level target depth value compression rate.

[0086] A thirteenth embodiment may include the method according to the twelfth embodiment, wherein the conversion function fM (x) also includes a third parameter and a fourth parameter, wherein the third parameter depends at least on the difference D between the depth weight value corresponding to the compression rate of the Mth level target depth value and the depth weight value corresponding to the compression rate of the M+1th level target depth value, and the fourth parameter depends at least on D, c1 and c2.

[0087] A fourteenth embodiment may include the method according to the thirteenth embodiment, wherein when N is equal to 2, the conversion function f M (x) can be realized according to the following formula: , where the third parameter is 1 and the fourth parameter is 0.

[0088] In a fifteenth embodiment, a head mounted display device includes: a tracking unit for detecting a focus area of ​​a user's eyes; a depth map processing unit including a memory storing computer instructions, which, when executed by one or more processors, executes a method as in one of the first to fourteenth embodiments to output a reconfigured depth map; a 3D engine for generating a 3D view according to the reconfigured depth map and the 2D image; and a display unit for displaying the generated 3D stereoscopic image.

[0089] In a sixteenth embodiment, an image processing device includes: a data receiving module configured to receive a 2D image, a depth map corresponding to the 2D image, and information about a focus area of ​​a user's eyes, wherein the depth map includes original depth values ​​representing pixels of the 2D image; a focus area analysis unit configured to calculate the distance between a pixel in the 2D image or the depth map and a center of the focus area; and a depth scaling module configured to reconfigure the depth map of the 2D image by compressing or amplifying original depth values ​​of pixels in at least one distance range according to a specified depth value compression rate or amplification rate based on the calculated distance between the pixel and the center of the focus area, so as to weaken or enhance the 3D perception of the corresponding distance range area.

[0090] The seventeenth embodiment may include the apparatus according to the sixteenth embodiment, further comprising: a 3D rendering engine configured to receive the reconfigured depth map, and to generate a 3D view based on the reconfigured depth map and the 2D image using a 3D engine.

[0091] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices, equipment, and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0092] By studying the drawings, the disclosure and the appended claims, those skilled in the art will be able to understand and implement variations to the disclosed embodiments in practicing the claimed subject matter. In the claims, the words "A and / or B" mean A, B, or A and B, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality; the words "first", "second", "third", "fourth" are used only to distinguish elements or steps and do not indicate the order of the elements or steps. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. An image processing method, comprising: Receive 2D images, A depth map corresponding to the 2D image is received, wherein: The depth map includes original depth values ​​of corresponding pixels of the 2D image pixels; receiving information about the focus area of ​​the eye; Calculate the distance between the pixel in the 2D image or depth map and the center of the focus area; reconfiguring the depth map corresponding to the 2D image by compressing or amplifying original depth values ​​of pixels in at least one distance range according to a specified depth value compression rate or amplification rate based on the calculated distance between the pixel and the center of the focus area, so as to weaken or enhance the 3D perception of the corresponding distance range area; as well as The reconfigured depth map is output for generating a 3D view based on the 2D image.

2. The method according to claim 1, wherein: The original depth value of the pixel has a first value range, and compressing or amplifying the original depth value of the pixel includes converting the original depth value of the pixel into a compressed depth value or an amplified depth value within a second value range according to a specified depth value compression rate or amplification rate.

3. The method according to claim 2, wherein: When the original depth value is magnified by a magnification greater than 1, the first value range is a subset of the second value range, and the second value range interval is proportionally magnified relative to the first value range according to the specified depth value magnification.

4. The method according to claim 2, wherein: When the depth value is compressed at a compression ratio less than 1, the second value range is a subset of the first value range, and the second value range interval is proportionally reduced relative to the first value range according to the specified depth value compression ratio.

5. The method according to claim 3 or 4, wherein: The median of the second numerical range is the same as the median of the first compressed or enlarged range, or the median of the second numerical range is different from the median of the first numerical range.

6. The method according to claim 5, wherein: For a given depth value compression or magnification ratio, the change in converting the original depth value of a pixel into a compressed or magnified depth value within the second value range is non-linear.

7. The method according to claim 5, wherein: For a specified depth value compression or magnification ratio, the change in the original depth value of the pixel into the compressed or magnified depth value within the second value range is linear. 8 . The method according to claim 6 , further comprising, for the 2D image pixels within the distance range, specifying a lowest first-level target depth value compression rate and a highest Nth-level target depth value compression rate of the pixels.

9. The method according to claim 8, further comprising: According to a desired degree of 3D perception, assigning depth weights to the 2D image pixels within the distance range to indicate a degree of compression of pixels at a distance x from the center of the focus region, so that starting from the center of the focus region, as the distance of the pixels increases, the depth value compression rate of the 2D image pixels increases from a first level target depth value compression rate to an Nth level target depth value compression rate; Here, a depth weight of 1 is assigned to pixels having a first-level target depth value compression rate, and a depth weight of 0 is assigned to pixels having an Nth-level target depth value compression rate.

10. The method according to claim 9, further comprising specifying that the 2D image pixels within the distance range have target depth value compression rates from level 2 to level N-1, wherein any level M target depth value compression rate is assigned a corresponding depth weight value between 0 and 1, and wherein at least some of the 2D image pixels within the distance range have corresponding depth weights from level 1 to level N, respectively.

Images