Method and device for processing three-dimensional image, storage medium and computing equipment
Patent Information
- Application Number
- CN202380010854.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-22
- Publication Date
- 2025-06-06
AI Technical Summary
When viewing naked-eye 3D pictures, recording 3D pictures is only valid in the initial position, resulting in perspective distortion of the picture when the human eye position is offset, affecting the immersion experience.
The head tracking device detects the update of the human eye position, obtains the coordinates of the virtual screen point on the virtual display, determines the tilt crop plane, and adjusts the virtual display to the display screen parallel to the display device by changing the projection matrix, obtains the front view and renders the output.
It reduces the picture distortion caused by human eye movement and improves the immersion and stability of the three-dimensional visual experience.
Smart Images
Figure CN120113233A_ABST
Abstract
Description
Method, device, storage medium, and computing device for processing three-dimensional images Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method, apparatus, computer-readable storage medium, and computing device for processing three-dimensional images. Background Art
[0002] When watching naked-eye 3D (3-dimension) images, since the 3D images are recorded at the initial position, the good 3D viewing effect can only be achieved when the human eye is facing the screen. When the human position shifts, the image perspective is still established based on the initial position, causing the image observed by the human eye to be distorted.
[0003] Summary of the Invention
[0004] In view of this, one of the objectives of the present invention is to provide a method, apparatus, computer-readable storage medium, and computing device for processing three-dimensional images.
[0005] In a first aspect of the present application, a method for processing a three-dimensional image is provided, wherein the three-dimensional image is a parallax image captured by a virtual camera and displayed on a display device, the method comprising:
[0006] When the head tracking device detects an update in the position of the human eye, the coordinates of the virtual screen point on the virtual display screen in the camera space are obtained; the virtual display screen is the area of interest viewed by the user; when the human eye is in the initial position, the image captured by the virtual camera at the installation position is consistent with the image captured by the human eye;
[0007] Determine the near clipping plane based on the coordinates of the virtual screen point in camera space;
[0008] Change the near clipping plane to the oblique clipping plane;
[0009] The inclined clipping plane is a plane that clips pixels outside the virtual display screen and is parallel to the virtual display screen;
[0010] Changing the projection matrix to adjust the virtual display screen to be parallel to the display screen of the display device using the changed projection matrix, thereby obtaining a front view of the virtual display screen. The projection matrix is used to represent the mapping relationship between the coordinates of the virtual screen point in the camera space and the coordinates in the image obtained by the human eye.
[0011] Render the front view and output the rendered result to a display device.
[0012] In a second aspect of the present application, a device for processing a three-dimensional image is provided. The three-dimensional image is a parallax image captured by a virtual camera and displayed on a display device. The device includes:
[0013] A coordinate acquisition module is used to obtain the coordinates of a virtual screen point on the virtual display screen in the camera space when the head tracking device detects an update in the position of the human eye. The virtual display screen is the area of interest viewed by the user. When the human eye is in the initial position, the image captured by the virtual camera at the installation position is consistent with the image captured by the human eye.
[0014] A near clipping plane determination module is configured to determine a near clipping plane based on the coordinates of a virtual screen point in camera space; change the near clipping plane to an oblique clipping plane; the oblique clipping plane is a plane parallel to the virtual display screen that clips pixels outside the virtual display screen;
[0015] A virtual view acquisition module is used to change the projection matrix so as to adjust the virtual display screen to be parallel to the display screen of the display device using the changed projection matrix to obtain a front view of the virtual display screen. The projection matrix is used to represent the mapping relationship between the coordinates of the virtual screen points in the camera space and the coordinates in the image obtained by the human eye.
[0016] The rendering module is used to render the front view and output the rendering result to the display device.
[0017] The third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method provided in the first aspect of the present application when the computer program is executed by a processor.
[0018] In a fourth aspect of the present application, a computing device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method provided in the first aspect of the present application are implemented.
[0019] Through the above solution, this application has at least the following beneficial effects:
[0020] When a change in the human eye position is detected, the coordinates of each point in the area of interest are obtained, a virtual display screen is set based on the area of interest, and an inclined clipping plane parallel to the virtual display screen is used as the near clipping plane. The projection matrix is then used to change the positions of all pixels on the virtual display screen to obtain a picture looking directly at the display screen. The rendered output obtains a perspective that conforms to the change in the human eye position, which reduces the picture distortion caused by human eye movement compared to conventional solutions. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0022] FIG1 is a schematic diagram of a display effect of the prior art.
[0023] FIG2 is a logic block diagram of a method for processing a three-dimensional image according to an embodiment of the present application.
[0024] FIG3 a is a schematic diagram of an initial position according to an embodiment of the present application.
[0025] FIG3 b is a schematic diagram showing a position change according to an embodiment of the present application without applying the method of the present application.
[0026] FIG3 c is a schematic diagram showing a position change and application of the method of the present application according to an embodiment of the present application.
[0027] FIG4 is a schematic diagram of a viewing cone of a binocular camera according to an embodiment of the present application.
[0028] FIG5 is a schematic diagram of an interface of an application image processing method according to an embodiment of the present application.
[0029] FIG6 is a schematic diagram of an apparatus for processing three-dimensional images according to an embodiment of the present application.
[0030] FIG7 is a schematic diagram of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION
[0031] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0032] As the quality of life improves, spiritual needs are also gradually increasing. When it comes to watching videos and playing games, more and more users are pursuing the ultimate three-dimensional visual experience. With years of technological accumulation, three-dimensional images are becoming increasingly refined, but there are still some problems that lead to insufficient immersive experience.
[0033] As shown in Figure 1, in a virtual 3D scene, when the user's eyes move from looking straight ahead at the screen (view a) to looking at the screen at an angle (view b), the perspective changes to reflect the eye's movement. However, because the change only accounts for eye movement, the resulting view c doesn't account for the fixed screen orientation. Consequently, in views a and b, the horizontal perspective line (the visible floor) of the 3D scene is aligned with the long edge of the display. However, in view c, the horizontal perspective line of the 3D scene is at an angle to the long edge of the display, resulting in a change in perspective and distortion to the naked eye. The spatial positions of objects in the 3D scene are offset compared to the real world, resulting in a lack of immersion.
[0034] To enhance the user's immersion in a 3D scene, with reference to FIG2 , this application proposes a method for processing a three-dimensional image, wherein the three-dimensional image is a parallax image captured by a virtual camera and displayed on a display device. The method includes:
[0035] S201, when the head tracking device detects an update in the position of the human eye, obtaining the coordinates of a virtual screen point on the virtual display screen in the camera space; the virtual display screen is the area of interest viewed by the user; when the human eye is in the initial position, the image captured by the virtual camera at the installation position is consistent with the image captured by the human eye;
[0036] S202, determining a near clipping plane based on the coordinates of the virtual screen point in the camera space;
[0037] S203, changing the near clipping plane to an inclined clipping plane; the inclined clipping plane is a plane that clips pixels outside the virtual display screen and is parallel to the virtual display screen;
[0038] S204. Modify the projection matrix to adjust the virtual display screen to be parallel to the display screen of the display device using the modified projection matrix, thereby obtaining a front view of the virtual display screen. The projection matrix is used to represent the mapping relationship between the coordinates of the virtual screen point in the camera space and the coordinates in the image obtained by the human eye.
[0039] S205: Render the front view, and output the rendering result to a display device.
[0040] It can be understood that in this article, 3D scenes or three-dimensional scenes are mainly divided into two types. One is the picture taken by a real camera in the real world and displayed on the monitor; the other is a virtual camera in the virtual world, which simulates the framing of the virtual scene and displays it on the monitor.
[0041] Through the above method, when the change of the human eye position is detected, the coordinates of each point in the area of interest are obtained. The virtual display screen is based on the area of interest, so the content on the virtual display screen represents the content to be observed. The inclined clipping plane parallel to the display screen is used as the near clipping plane, and then the projection matrix is used to change the position of all pixel points on the virtual display screen to obtain the picture facing the display screen, and the rendered output is obtained. Compared with conventional solutions, the picture distortion caused by the movement of the human eye is reduced.
[0042] In order to have a more intuitive understanding of the solution of the present application, reference may be made to FIG3a . FIG3a shows the initial position, that is, the user is facing the display. The left side is a top view, showing the positional relationship between the user and the display. The right side is the display screen observed by the user, and no position change occurs at this time.
[0043] Further, please refer to Figure 3b. The left side of the figure shows that the user has made a displacement compared to the display. At this time, if the camera is a virtual camera, the picture of the changed virtual camera position will be displayed as described in the figure. At this time, the picture has changed, but the solution of the present application is not adopted. Therefore, although the display picture is correct, the user who looks at it at an angle cannot see the correct picture effect, and no matter how to observe it, the perspective of the picture content has changed compared to the initial position (the checkerboard edge and the physical display form an angle).
[0044] If the solution of the present application is adopted at this time, you can refer to Figure 3c and use the dotted box in Figure 3b as the area of interest to perform a series of cropping and matrix transformation operations. The final output image is shown on the right side of Figure 3c. The effect image observed by the user is not distorted, and the same as the initial position is that the perspective in the 3D scene is still the same as the perspective of the physical display (the checkerboard and the display maintain no angle).
[0045] In any embodiment of the present application, when coordinates are involved, the following rules can be referred to: the center of the virtual display screen is the origin, the horizontal direction of the virtual display screen is the x-axis, the vertical direction is the y-axis, and the direction facing the camera is the z-axis.
[0046] In one embodiment, in step S201, obtaining the coordinates of a virtual screen point on the virtual display screen in the camera space includes:
[0047] Obtain the offset angle of the virtual screen point on the virtual display screen relative to the virtual camera;
[0048] The space conversion matrix is calculated based on the offset angle, and the coordinates of the virtual screen point in the camera space are obtained based on the world space coordinates of the virtual screen point and the space conversion matrix, so as to convert the world space coordinate system into the camera space coordinate system.
[0049] After the control conversion, it is easy to obtain the offset angle of the virtual screen point relative to the camera, which is beneficial for subsequent calculations.
[0050] In one embodiment, the following formula may be used for calculation:
[0051] In formula 1, θ x Indicates the offset angle of the virtual screen point in the x direction relative to the camera, (x0, y0, z0) represents the coordinates of the virtual screen point, and x0, y0, and z0 represent the coordinate values of the virtual screen point on the x, y, and z axes respectively.
[0052] In formula 2, θ y Indicates the offset angle of the virtual screen point in the x direction relative to the camera. x0 and z0 represent the coordinate values of the virtual screen point on the x and z axes respectively.
[0053] According to formula 1 and formula 2, the spatial transformation matrix M can be obtained as follows:
[0054] For the spatial transformation matrix M (i.e., Formula 3), where R x 、R y 、R z Represents the rotation matrix of x, y, and z axes respectively, M translation Represents the translation matrix, M represents the final spatial transformation matrix, where the default scaling factor is 1, which means the size does not change.
[0055] In formulas 1-3, the same letters represent the same parameters, and this application will not repeat them one by one.
[0056] In another embodiment, if the size of the picture is to be changed, 1 can be modified to a numerical value.
[0057] To simulate the effect of human observation, the camera can adopt a dual-camera design to achieve the parallax effect. Therefore, in this application, when the camera is a virtual camera, the two virtual cameras are set to move or rotate synchronously; the distance between the two virtual cameras is fixed and equal to or close to the interpupillary distance of the human eye; the virtual camera is used to simulate the RGB camera.
[0058] It is understandable that if the camera is a physical camera, two cameras can still be used as binocular cameras to collect materials. The two cameras of the binocular camera are fixedly connected and move or rotate at the same time, and the distance between the two cameras is fixed and equal to or close to the pupil distance of the human eye.
[0059] It is understandable that the interpupillary distance of the human eye varies depending on the user. In actual products, a data corresponding to a percentile can be selected as the product's adjustable range. For example, in one embodiment, the range can be 14-20mm. However, it is possible that some users' interpupillary distances fall outside this range. Therefore, if the distance between the two cameras is equal to or close to the interpupillary distance of the human eye, it should fall within the scope of protection of this application.
[0060] In one embodiment, a black and white camera may be used to capture images in a special style, such as a 3D image in an ink painting style.
[0061] Based on any of the above embodiments, the aspect ratio of the virtual display screen is the same as the display aspect ratio of the display device; or the angle between the normal of the virtual display screen and the virtual camera is an acute angle; or the virtual display screen is within the range that can be captured by the two virtual cameras.
[0062] By using the above method, setting the aspect ratio of the virtual display screen to be the same as the aspect ratio of the display device can further increase the screen-to-body ratio and prevent image distortion caused by the display device stretching due to different display ratios.
[0063] When the angle between the normal of the virtual display screen and the virtual camera is acute, sufficient image information can be displayed on the virtual display screen. Especially when the angle is 0, the virtual display screen can display the most image information, avoiding the inability to crop the image due to insufficient pixels.
[0064] In this application, in step S203, the near clipping plane is changed to an oblique clipping plane. This is to clip points outside the virtual screen. As shown in Figure 4, the viewing cone of camera C' includes areas A and B. After setting the oblique projection plane, area B is clipped, leaving only area A. C represents the viewing cone of the other camera. When using binocular cameras, each camera can correspond to a viewing cone, and each viewing cone can correspond to a coordinate system. P represents the symmetry midline of the binocular cameras.
[0065] Combining Formula 1-Formula 3, in one embodiment, the reasoning process of the oblique projection is as follows: set the oblique clipping plane to Plane, the near clipping plane and the far clipping plane of the camera to near and far respectively, then satisfy: near = M4 + M3, far = M4 - M3. Where M4 and M3 represent the projection matrix M respectively. frustum The fourth and third rows of , and so on for the other rows, so after replacing the near clipping plane, Plane=M4'+M3'.
[0066] Since M4 needs to maintain the depth information of the camera space and cannot be changed, M4' = M4. The M3 vector needs to be modified, so M3' = Plane - M4. The far clipping plane after replacement is far' = M4 - M3' = 2M4 - Plane.
[0067] On this basis, in order to ensure that the depth accuracy is not affected, after step S203, that is, after the near clipping plane is changed to the inclined clipping plane, the method may further include:
[0068] The far clipping plane is adjusted using a correction factor. The correction factor is used to reduce the angle between the near clipping plane and the far clipping plane. The far clipping plane is the plane that defines the farthest distance that the virtual camera can render.
[0069] Through the above scheme, the correction factor is used to adjust the far clipping plane and reduce the angle between the near clipping plane and the far clipping plane so that the near clipping plane and the far clipping plane tend to be parallel, avoiding the change of depth accuracy caused by the picture change.
[0070] In some embodiments, once the camera is installed, its position cannot be changed, and the single image obtained cannot be changed.
[0071] Therefore, in this application, the camera is set at a fixed position relative to the virtual display screen, and the method may further include:
[0072] The position of the virtual screen point is updated by calling a custom interface so that the virtual camera can obtain an image of the perspective after the human eye moves; the interface is used to receive the human eye position feedback from the head tracking device.
[0073] Through the above scheme, an interface for modifying virtual screen points is provided. On the basis of meeting the requirements in step S201, by arbitrarily defining or changing the position of the virtual screen point, the resulting picture effect is equivalent to the camera moving. By using this image processing method for a single image, different 3D images from multiple perspectives can be viewed, which increases the viewing diversity and resource richness, and improves the viewing experience when the camera cannot be moved.
[0074] Based on any of the above embodiments, in one embodiment, step S205 of rendering the front view may include:
[0075] Convert the coordinates of the virtual screen point into two-dimensional coordinates;
[0076] The value of the two-dimensional coordinate is enlarged so that the virtual screen point corresponding to the front view can cover the entire display screen of the display device.
[0077] Accordingly, the present application provides a specific method that is not intended to limit the scope of protection of the present application, as follows:
[0078] As shown in view b of Figure 5, let the figure formed by points ABCD be similar to the border formed by points EFGH. Specifically, let P A 、P B 、P C 、P D Represent the coordinates of the four corners of the virtual display screen in the camera space, let P A1 、P B1 、P C1 、P D1 Represents the coordinates of the four corners of the virtual display screen in the projection space, and P A1 =M frustum *P A (Points B, C, D and so on), M frustum is the projection matrix. To ensure that the plane of the virtual display screen is parallel to the display screen, Formula 4 must be satisfied:
[0079] In Formula 4, x represents the x-axis, y represents the y-axis, and w represents the depth.
[0080] That is, change the projection matrix (M frustum ), so that the projection points of A, B, C, and D satisfy the above formula. For any camera, assuming its parameters include field of view FOV, aspect ratio Aspect, near clipping screen Near, far clipping plane Far, where aspect ratio Aspect = NearHeight / NearWidth = FarHeight / FarWidth = screen height / screen width, then the most basic projection matrix (M frustum )for:
[0081] The analysis of the zero elements in the matrix of formula 5 is as follows:
[0082] Since the projection onto the plane is two-dimensional, M13 (first row, third column, and so on) and M14 in row M1 (first row) will not affect the relationship between the coordinate points x and y, which is equivalent to adding a constant. The presentation effect is that when M13>0, the overall image moves toward the positive direction of the x-axis, and when M14>0, the overall image moves toward the negative direction of the x-axis; M12>0 will cause the x-coordinate point to increase as y increases, and the overall image will be distorted.
[0083] The same is true for the second row of the projection matrix. M23 and M24 do not affect the relationship between the coordinate points x and y, which is equivalent to adding a constant. The presentation effect is that when M23>0, the entire image moves toward the positive direction of the y-axis, and when M24>0, the entire image moves toward the negative direction of the y-axis. M21>0 will cause the y coordinate point to increase as x increases, and the entire image will be distorted.
[0084] The M4 row changes the depth information of the entire image. The second coordinate point corresponding to M44 is a constant 1. Therefore, no matter how much M44 is, it is equivalent to adding a constant, and the image presents an overall zooming effect. M41, M42, and M43 comprehensively change the depth information of all patterns in the viewing cone.
[0085] The M3 operation is the same as that shown in steps S202 and S203 , except that the shape of the clipping plane is changed, and points outside the virtual screen can be clipped to obtain the image viewed by the first virtual person.
[0086] In order to make the depth information of the virtual display screen composed of four points (ABCD) the same, in one embodiment, the formula P is used. A1 (w) = P B1 (w) = P C1 (w) = P D1 (w) Unified depth information.
[0087] Assume that in formula 5, M41 and M42 are unknowns, and then the results after analysis are as shown in formulas 6 and 7; when the depth information is the same, that is, w is the same, the analysis of P A1 (x) = P B1 (x)(or P C1 (x) = P D1 (x)), the parameter results of M12 can be obtained as shown in Formula 8. Similarly, the parameter results of M21 can be obtained as shown in Formula 9:
[0088] Modifying the projection matrix can obtain the new points PA1, PB1, PC1, and PD1 after projection, and the two-dimensional coordinate point P can be obtained after standardization. A2 、P B2 、P C2 、P D2 , where P A2 (x) = P A1 (x) / P A1 (w) / 2+0.5,P A2 (y) = P A1 (y) / P A1 (w) / 2+0.5 (points B, C, and D are the same), extract the camera's rendering texture, and in the process of rendering to the material, amplify it through formulas 9 and 10 to get P A2 =[0,1],P B2 =[0,0],P C2 =[1,1],P D2 = [1,0] means the virtual screen point is rendered on the entire screen. For details, please refer to formula 10 and formula 11: u′=P A2(x)+(P C2 -P A2 (x))*u(Formula 10); v′=P A2 (y)+(P B2 -P A2 (y))*v(Formula 11).
[0089] The texture belongs to plane information, u and v represent parameters of the texture in two directions before transformation, and u' and v' represent parameters of the texture in two directions after transformation.
[0090] Embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.
[0091] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform the corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
[0092] Computers suitable for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to such mass storage devices to receive data from them or to transmit data to them, or both. However, a computer does not necessarily have such devices. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.
[0093] Corresponding to the method, with reference to FIG6 , the present application further provides an apparatus 60 for processing a three-dimensional image. The three-dimensional image is a parallax image captured by a virtual camera and displayed on a display device. The apparatus 60 may include:
[0094] The coordinate acquisition module 601 is configured to obtain the coordinates of a virtual screen point on the virtual display screen in the camera space when an update of the eye position is detected by the head tracking device; the virtual display screen is the area of interest viewed by the user; when the eye is in the initial position, the image captured by the virtual camera at the installation position is consistent with the image captured by the eye;
[0095] A near clipping plane determining module 602 is configured to determine a near clipping plane based on the coordinates of the virtual screen point in the camera space; change the near clipping plane to an oblique clipping plane; the oblique clipping plane is a plane parallel to the virtual display screen that clips pixels outside the virtual display screen;
[0096] A virtual view acquisition module 603 is configured to modify a projection matrix to adjust the virtual display screen to be parallel to the display screen of the display device using the modified projection matrix, thereby obtaining a front view of the virtual display screen. The projection matrix represents the mapping relationship between the coordinates of virtual screen points in the camera space and the coordinates in the image obtained by the human eye.
[0097] The rendering module 604 is used to render the front view and output the rendering result to a display device.
[0098] On this basis, when the camera is a virtual camera, the two virtual cameras are set to move or rotate synchronously; the distance between the two virtual cameras is fixed and equal to or close to the pupil distance of the human eye; the virtual camera is used to simulate the RGB camera.
[0099] In another embodiment, the aspect ratio of the virtual display screen is the same as the display aspect ratio of the display device; or the angle between the normal of the virtual display screen and the virtual camera is an acute angle; or the virtual display screen is within the range that can be captured by the two virtual cameras.
[0100] In another embodiment, the device 60 may further include: an angle correction module, which is used to adjust the far clipping plane using a correction factor after changing the near clipping plane to an inclined clipping plane, and the correction factor is used to reduce the angle between the near clipping plane and the far clipping plane. The far clipping plane is a plane that defines the farthest distance that the virtual camera can render.
[0101] In yet another embodiment, the camera is disposed at a fixed position relative to the virtual display screen, and the apparatus 60 may further include:
[0102] The position update module is used to update the position of the virtual screen point by calling a custom interface so that the virtual camera can obtain an image of the perspective after the human eye moves; the interface is used to receive the human eye position feedback from the head tracking device.
[0103] In one embodiment, the rendering module 604 may include:
[0104] A coordinate conversion submodule, used to convert the coordinates of the virtual screen point into two-dimensional coordinates;
[0105] The magnification submodule is used to magnify the value of the two-dimensional coordinate so that the virtual screen point corresponding to the front view can cover the entire display screen of the display device.
[0106] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.
[0107] Accordingly, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method of any embodiment are implemented.
[0108] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.
[0109] 7 , the present application further provides a computing device 70 , which may include a memory 701 , a processor 702 , and a computer program stored in the memory 701 and executable on the processor 702 , wherein the processor 702 implements the steps of any embodiment method when executing the program.
[0110] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.
[0111] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device that includes the element.
[0112] The above is a detailed introduction to the methods and devices provided in the embodiments of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the methods and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.
Claims
1. A method for processing a three-dimensional image, wherein the three-dimensional image is a parallax image captured by a virtual camera and displayed on a display device, characterized in that: The method comprises: When the head tracking device detects that the position of the human eye is updated, the coordinates of the virtual screen point on the virtual display screen in the camera space are obtained; the virtual display screen is the area of interest viewed by the user; when the human eye is in the initial position, the image captured by the virtual camera at the installation position is consistent with the image obtained by the human eye; Determine a near clipping plane based on the coordinates of the virtual screen point in the camera space; The near clipping plane is changed to an inclined clipping plane; the inclined clipping plane is a plane that clips pixels outside the virtual display screen and is parallel to the virtual display screen; Changing the projection matrix, so as to adjust the virtual display screen to be parallel to the display screen of the display device by using the changed projection matrix, and obtaining a front view of the virtual display screen, wherein the projection matrix is used to represent the mapping relationship between the coordinates of the virtual screen point in the camera space and the coordinates in the image obtained by the human eye; Render the front view, and output the rendering result to the display device.
2. The method according to claim 1, characterized in that The two virtual cameras are configured to move or rotate synchronously; the distance between the two virtual cameras is fixed and is equal to or close to the pupil distance of a human eye; and the virtual camera is used to simulate an RGB camera.
3. The method according to claim 1, characterized in that The aspect ratio of the virtual display screen is the same as the display aspect ratio of the display device; or the angle between the normal of the virtual display screen and the virtual camera is an acute angle; or the virtual display screen is within the range that can be photographed by two virtual cameras.
4. The method according to claim 1, characterized in that After changing the near clipping plane to an inclined clipping plane, the method further includes: The far clipping plane is adjusted using a correction factor, where the correction factor is used to reduce an angle between the near clipping plane and the far clipping plane, and the far clipping plane is a plane defining a maximum distance that the virtual camera can render.
5. The method according to claim 1, characterized in that The virtual camera is set at a fixed position of the virtual display screen, and the method further includes: The position of the virtual screen point is updated by calling a custom interface so that the virtual camera obtains an image of the perspective after the human eye moves; the interface is used to receive the human eye position fed back by the head tracking device.
6. The method according to claim 1, characterized in that Rendering the front view includes: Converting the coordinates of the virtual screen point into two-dimensional coordinates; The value of the two-dimensional coordinate is enlarged so that the virtual screen point corresponding to the front view can cover the entire display screen of the display device.
7. A device for processing a three-dimensional image, wherein the three-dimensional image is a parallax image captured by a virtual camera and displayed on a display device, characterized in that: The device comprises: A coordinate acquisition module, used to obtain the coordinates of a virtual screen point on a virtual display screen in the camera space when the head tracking device detects that the position of the human eye is updated; the virtual display screen is the area of interest viewed by the user; when the human eye is in the initial position, the image captured by the virtual camera at the installation position is consistent with the image captured by the human eye; A near clipping plane determination module is used to determine a near clipping plane based on the coordinates of the virtual screen point in the camera space; change the near clipping plane to an inclined clipping plane; the inclined clipping plane is a plane that clips the pixels outside the virtual display screen and is parallel to the virtual display screen; A virtual view acquisition module, used to change a projection matrix, so as to adjust the virtual display screen to be parallel to the display screen of the display device by using the changed projection matrix, and obtain a front view of the virtual display screen, wherein the projection matrix is used to represent a mapping relationship between the coordinates of the virtual screen point in the camera space and the coordinates in the image obtained by the human eye; A rendering module is used to render the front view and output the rendering result to the display device.
8. The device according to claim 7, characterized in that The two virtual cameras are configured to move or rotate synchronously; the distance between the two virtual cameras is fixed and is equal to or close to the pupil distance of a human eye; and the virtual camera is used to simulate an RGB camera.
9. The device according to claim 7, characterized in that The aspect ratio of the virtual display screen is the same as the display aspect ratio of the display device; or the angle between the normal of the virtual display screen and the virtual camera is an acute angle; or the virtual display screen is within the range that can be photographed by two virtual cameras.
10. The device according to claim 7, characterized in that The device also includes: The angle correction module is used to adjust the far clipping plane by using a correction factor after changing the near clipping plane to an inclined clipping plane, wherein the correction factor is used to reduce the angle between the near clipping plane and the far clipping plane, and the far clipping plane is a plane defining the farthest distance that the virtual camera can render.
11. The device according to claim 7, characterized in that The virtual camera is arranged at a fixed position of the virtual display screen, and the device further comprises: The position of the virtual screen point is updated by calling a custom interface so that the virtual camera obtains an image of the perspective after the human eye moves; the interface is used to receive the human eye position fed back by the head tracking device.
12. The device according to claim 7, characterized in that The rendering module includes: A coordinate conversion submodule, used for converting the coordinates of the virtual screen point into two-dimensional coordinates; The enlargement submodule is used to enlarge the value of the two-dimensional coordinate so that the virtual screen point corresponding to the front view It can cover the entire display screen of the display device.
13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
14. A computing device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.