Data processing method and related device

Abstract

The invention discloses a data processing method, and the method comprises the steps: obtaining a first image and a watching parameter, the first image being a 3D image, and the watching parameter comprising the information of a virtual screen; when the first image is displayed according to the watching parameters, obtaining a second image through rendering according to the first image and the watching parameters based on the condition that the parallax of the first image on the virtual screen meets a preset condition; the second image is an image which belongs to the same 3D video stream as the first image and is behind the first image, and the parallax of the second image on the virtual screen is different from the parallax of the first image on the virtual screen when the second image is displayed. According to the method and the device, the parallax adjustment of the image during the display of the 3D video is realized.

Description

Technical Field

[0001] This application relates to the field of terminal equipment, and more particularly to a data processing method and related apparatus. Background Technology

[0002] Head-mounted display devices can enhance or expand the real-world scene seen by users using additional information generated by computers, greatly changing the way humans interact with computers or the outside world. These devices integrate multiple technologies from different research fields and are quickly being applied to entertainment, scientific research, simulation training, telemedicine and other fields.

[0003] When watching 3D videos, because there is a certain distance between the two eyes, the images seen by each eye are slightly different. This visual difference is called binocular parallax. The brain fuses the two differing images to obtain the depth relationship in three-dimensional space, thus forming a sense of spatial depth.

[0004] When watching 3D videos, the greater the degree of out-of-screen and in-screen movement (that is, the greater the absolute value of parallax), the stronger the 3D stereoscopic effect will be. However, if it exceeds a certain range (or exceeds a certain range for a long time), it will lead to visual fatigue or difficulty in image fusion.

[0005] Therefore, a method for parallax adjustment of images during 3D video display is proposed. Summary of the Invention

[0006] In a first aspect, this application provides a data processing method, the method comprising: acquiring a first image and viewing parameters, the first image being a 3D image, the viewing parameters including information of a virtual screen; when displaying the first image according to the viewing parameters, based on the parallax of the first image on the virtual screen satisfying a preset condition, obtaining a second image by rendering according to the first image and the viewing parameters; the second image being an image belonging to the same 3D video stream as the first image and following the first image, and the parallax of the second image on the virtual screen being different from the parallax of the first image on the virtual screen when the second image is displayed.

[0007] When watching 3D videos, the greater the degree of in-screen and out-of-screen movement (i.e., the greater the absolute value of parallax), the stronger the 3D effect. However, if this exceeds a certain range (or exceeds a certain range for an extended period), it can lead to visual fatigue or difficulties in image fusion. In this embodiment, when displaying a 3D image, it is determined whether the parallax of the displayed image meets a preset condition. If the parallax meets the preset condition, the parallax of the displayed image is adjusted, and an image with a different parallax than the previously displayed image is displayed. For example, when the parallax of the first image on the virtual screen meets the preset condition, a second image with a different parallax than the first image can be rendered, thereby realizing the parallax adjustment of the image when displaying 3D videos.

[0008] In one possible implementation, the parallax on the virtual screen based on the first image satisfying a preset condition includes: the parallax on the virtual screen based on at least one display pixel of the first image satisfying a preset condition.

[0009] In one possible implementation, the condition that the parallax of the first image on the virtual screen meets a preset condition includes: the parallax of the first image on the virtual screen is not within the safe parallax range; and the parallax of the second image on the virtual screen is within the safe parallax range.

[0010] In one possible implementation, the 3D video frame stream can be traversed and the disparity value range of each frame pair (each pair of the same name, that is, the same point on the left and right eye images) can be calculated to determine whether it is within the safe disparity range. If the disparity on the virtual screen is not within the safe disparity range, the disparity of the displayed image can be adjusted.

[0011] In one possible implementation, the method further includes: acquiring data on a user's eye state while viewing the first image; and when the data indicates that the user's eyes are fatigued, the parallax of the first image on the virtual screen satisfies a preset condition.

[0012] In one possible implementation, the absolute value of the parallax of the second image on the virtual screen is less than the absolute value of the parallax of the first image on the virtual screen.

[0013] In one possible implementation, the method further includes: acquiring data on a user's eye state while viewing the first image; when the data indicates that the user's eyes are in a healthy state (i.e., not fatigued), the parallax of the first image on the virtual screen satisfies a preset condition; and the absolute value of the parallax of the second image on the virtual screen is greater than the absolute value of the parallax of the first image on the virtual screen.

[0014] There are significant individual differences in image fusion ability. People with strong image fusion ability can watch videos with a strong 3D feel, while in extreme cases, some people have such weak image fusion ability that they cannot watch 3D videos at all. This application can collect information about the user's eye state when watching 3D videos and adjust the parallax. When the eyes are fatigued, the absolute value of the parallax can be reduced to ensure the user's visual comfort. When the eyes are healthy, the absolute value of the parallax can be increased to enhance the 3D stereoscopic effect.

[0015] In one possible implementation, the method further includes: when displaying the first image according to the viewed parameters, receiving an adjustment instruction for the parallax obtained from user input, and determining that the parallax of the first image on the virtual screen meets a preset condition.

[0016] In one possible implementation, the user input is at least one of the following: input via user gesture, input via keypad, or input via UI page.

[0017] In this embodiment, users can actively adjust the parallax according to their preferences or comfort through gestures / buttons / UI interaction.

[0018] In one possible implementation, the step of obtaining a second image by rendering based on the first image and the viewing parameters includes: obtaining a new viewing angle based on an adjustment amount of the parallax of the first image on the virtual screen; and obtaining the second image by rendering based on the new viewing angle, the first image, and the viewing parameters.

[0019] In one possible implementation, obtaining a second image by rendering based on the first image and the viewing parameters includes: obtaining a second image by rendering based on the first image, the corresponding depth map, and the viewing parameters.

[0020] For example, a second image can be obtained by rendering based on the first image, its corresponding depth map, and the viewing parameters. By introducing a depth-based fast image rendering technique, information loss and distortion problems caused by cropping are avoided, and better parallax adjustment effects can be achieved.

[0021] Secondly, this application provides a data processing apparatus, the apparatus comprising:

[0022] An acquisition module is used to acquire a first image and viewing parameters, wherein the first image is a 3D image and the viewing parameters include information about the virtual screen.

[0023] The processing module is configured to, when displaying the first image according to the viewing parameters, render a second image based on the parallax of the first image on the virtual screen satisfying a preset condition, according to the first image and the viewing parameters; the second image is an image belonging to the same 3D video stream as the first image and following the first image, and the parallax of the second image on the virtual screen when displaying the second image is different from the parallax of the first image on the virtual screen.

[0024] In one possible implementation, the parallax on the virtual screen based on the first image satisfying a preset condition includes: the parallax on the virtual screen based on at least one display pixel of the first image satisfying a preset condition.

[0025] In one possible implementation, the condition that the parallax of the first image on the virtual screen meets a preset condition includes: the parallax of the first image on the virtual screen is not within the safe parallax range; and the parallax of the second image on the virtual screen is within the safe parallax range.

[0026] In one possible implementation, the acquisition module is further configured to: acquire data on the user's eye state while viewing the first image; and when the data indicates that the user's eyes are fatigued, the parallax of the first image on the virtual screen satisfies a preset condition.

[0027] In one possible implementation, the 3D video frame stream can be traversed and the disparity value range of each frame pair (each pair of the same name, that is, the same point on the left and right eye images) can be calculated to determine whether it is within the safe disparity range. If the disparity on the virtual screen is not within the safe disparity range, the disparity of the displayed image can be adjusted.

[0028] In one possible implementation, the absolute value of the parallax of the second image on the virtual screen is less than the absolute value of the parallax of the first image on the virtual screen.

[0029] In one possible implementation, the acquisition module is further configured to: acquire data on the user's eye state while viewing the first image; when the data indicates that the user's eyes are in a healthy state (i.e., not fatigued), the parallax of the first image on the virtual screen satisfies a preset condition; and the absolute value of the parallax of the second image on the virtual screen is greater than the absolute value of the parallax of the first image on the virtual screen.

[0030] In one possible implementation, the processing module is further configured to: when displaying the first image according to the viewed parameters, receive an adjustment instruction for the parallax obtained from the user's input, and determine that the parallax of the first image on the virtual screen meets a preset condition.

[0031] In one possible implementation, the user input is at least one of the following: input via user gesture, input via keypad, or input via UI page.

[0032] In one possible implementation, the processing module is specifically configured to: obtain a new viewing angle based on the amount of parallax adjustment of the first image on the virtual screen; and obtain a second image by rendering based on the new viewing angle, the first image, and the viewing parameters.

[0033] In one possible implementation, obtaining a second image by rendering based on the first image and the viewing parameters includes: obtaining a second image by rendering based on the first image, the corresponding depth map, and the viewing parameters.

[0034] A third aspect of this application provides a data processing apparatus, which may include a processor and a memory coupled together. The memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the first aspect or any implementation thereof is implemented. For details regarding the steps of the various possible implementations of the first aspect executed by the processor, please refer to the first aspect; further details will not be repeated here.

[0035] The fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method of any implementation of the first aspect described above.

[0036] The fifth aspect of this application provides a circuit system including a processing circuit configured to perform the method of any implementation of the first aspect described above.

[0037] The sixth aspect of this application provides a computer program product that, when run on a computer, causes the computer to perform any implementation of the first aspect described above.

[0038] A seventh aspect of this application provides a chip system including a processor for supporting a server or threshold value acquisition device in implementing the functions involved in any implementation of the first aspect described above, such as transmitting or processing data and / or information involved in the methods described above. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the server or communication device. This chip system may be composed of chips or may include chips and other discrete devices.

[0039] The beneficial effects of the second to seventh aspects mentioned above can be referred to the introduction of the first aspect above, and will not be repeated here. Attached Figure Description

[0040] Figure 1 A schematic diagram of an application architecture provided for an embodiment of this application;

[0041] Figure 2 A schematic diagram of a system architecture is provided for an embodiment of this application;

[0042] Figures 3 to 13 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0043] Figure 14 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0044] Figure 15 This is a schematic diagram of the structure of an execution device provided in an embodiment of this application. Detailed Implementation

[0045] The embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. As those skilled in the art will recognize, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0046] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or apparatus. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological or logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved.

[0047] Figure 1 A schematic diagram of a head-mounted display device 100 worn by a user 102 is shown. The head-mounted display device 100 can be used to display augmented reality images and physical objects in a real-world background scene. The head-mounted display device 100 may include a frame 104 (which may also be referred to as a glasses frame or glasses stand in this embodiment) for positioning the device at a target viewing position relative to the eyes of the user 102.

[0048] Figure 2 Show Figure 1 A schematic diagram of a head-mounted display device 100, as shown below. Figure 2 As shown, the head-mounted display device 100 includes a right-eye display system 200a and a left-eye display system 200b. The right-eye display system 200a or the left-eye display system 200b can be used to display virtual images to the user while allowing the user to view the real environment. The right-eye display system 200a may include a right-eye display module, and the left-eye display system 200b may include a left-eye display module. The head-mounted display device 100 also includes a middle shell and temples (or ear supports) 206. The middle shell is used to fix the left-eye display module, the right-eye display module, and the temples 206. The head-mounted display device 100 also includes a front shell connected to the middle shell. The front shell is located on the outer surface of the head-mounted display device and is used to protect the left-eye display module and the right-eye display module. It should be understood that the front shell can be a transparent light-shielding visor.

[0049] also, Figure 2A microphone 202 is schematically shown that can be used to output acoustic information to a user. Such acoustic information can take any suitable form, including but not limited to computer-generated speech output in an appropriate language (such as user-selected), tones or other sounds not specific to any language, and / or any other suitable sound. In some embodiments, other types of output may be provided by the head-mounted display device 100, such as haptic / touch output.

[0050] The left-eye display system 200b and the right-eye display system 200a can be positioned relative to the viewing position of the eyes via one or more fastening mechanisms of the frame 104. For example, as Figure 2 As shown, the frame 104 can be supported by the user's ear via the ear support 206 and by the user's nose via the bridge of the nose 208 to reduce slippage of the frame 104. It will be understood that... Figure 2 The supports shown (such as ear supports 206, nose supports, and bridge of the nose 208) are exemplary in nature, and the see-through display system (right eye display system 200a and left eye display system 200b) of the head-mounted see-through display device can be positioned at the viewing location via any suitable mechanism. For example, additional supports can be used, and / or Figure 2 One or more of the supports shown may be removed, replaced, and / or expanded to position the perspective display system at the viewing location. Furthermore, the perspective display system may be positioned at the viewing location by means other than supports that are physically in contact with the user; this application is not limiting.

[0051] It should be understood that the left-eye display module and the right-eye display module in this embodiment can be referred to as display device modules.

[0052] Head-mounted display devices can enhance or expand the real-world scene seen by users using additional information generated by computers, greatly changing the way humans interact with computers or the outside world. These devices integrate multiple technologies from different research fields and are quickly being applied to entertainment, scientific research, simulation training, telemedicine and other fields.

[0053] When watching 3D videos, because there is a certain distance between the two eyes, the images seen by each eye are slightly different. This visual difference is called binocular parallax. The brain fuses the two differing images to obtain the depth relationship in three-dimensional space, forming a sense of spatial depth. Objects with large parallax appear closer, while objects with small parallax appear farther away (e.g., objects with large parallax appear closer). Figure 1 The stereoscopic effect when watching 3D videos mainly comes from the parallax created by the left and right eyes; different parallaxes will produce out-of-screen / in-screen effects (such as...). Figure 2 If the convergence point of 3D objects falls exactly on the display plane, then there is zero parallax.

[0054] When watching 3D videos, the greater the degree of out-of-screen and in-screen movement, the stronger the 3D effect. However, if it exceeds a certain range, it can lead to visual fatigue or difficulties in image fusion (e.g., Figure 3 The ability to fuse images varies significantly from person to person. People with strong 3D abilities can watch videos with a strong sense of 3D, while in extreme cases, some people have such weak 3D abilities that they cannot watch 3D videos at all. Our current goal is: for general users, how can we ensure a good stereoscopic effect while avoiding visual fatigue and guaranteeing a comfortable viewing experience?

[0055] When users view their own 3D photos / videos using traditional 3D playback devices (such as Vision Pro) Figure 7 However, the 3D effect is often not adjustable, significantly diminishing the viewing experience. This is because videos shot from random camera angles rarely achieve a good 3D effect; to improve the viewing experience, it's necessary to set up a real-time adjustment mechanism for the in-screen and out-of-screen effects (such as...). Figure 8 ).

[0056] Please see Figure 3 , Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Figure 3 As shown, the data processing method includes the following steps 301-302.

[0057] Step 301: Obtain a first image and viewing parameters. The first image is a 3D image, and the viewing parameters include information about the virtual screen.

[0058] In one possible implementation, a first image can be acquired, which can be a 3D image, such as a frame from a 3D video captured by a camera. Specifically, during acquisition, an RGB frame sequence (including the first image, which can be a stereo video or photo stream) can be captured by a camera, a corresponding depth map can be acquired by a TOF sensor, and pose data (also referred to as the viewing angle in this embodiment) can be acquired by an inertial sensor. Additionally, shooting parameters such as baseline, focal length, and principal point can be acquired. It should be understood that the aforementioned image, depth map, and pose data should be time-aligned (or synchronized).

[0059] When displaying the first image (e.g., a 3D video frame including the first image) through a head-mounted device (e.g., AR device, VR device, MR device, XR device, etc.), it is also necessary to obtain viewing parameters. Viewing parameters include, but are not limited to, information about the virtual screen. When displaying the first image, the first image and the virtual screen can be displayed together. From the perspective of display effect, the positional relationship between the objects in the virtual screen and the first image (e.g., out-of-screen and in-screen) can present a 3D stereoscopic effect. For example, the greater the degree of out-of-screen and in-screen, the stronger the 3D stereoscopic effect will be.

[0060] The information for the virtual screen can include screen distance, size, etc.

[0061] In one possible implementation, data preprocessing can also be performed to achieve coarse adjustment of disparity. For example, preliminary binocular disparity measurement and correction can be performed, and binocular depth estimation can be performed to obtain its depth information.

[0062] For example, aligned RGB and ToF data can be input into a depth estimation fusion network to detect the temporal and spatial consistency of its output binocular disparity map. Temporal smoothing and depth completion can be performed on the inconsistencies, and filtering algorithms (such as WLS) can be used for processing (e.g., edge denoising, global smoothing, etc.).

[0063] Step 302: When displaying the first image according to the viewing parameters, based on the parallax of the first image on the virtual screen satisfying a preset condition, a second image is obtained by rendering according to the first image and the viewing parameters; the second image is an image that belongs to the same 3D video stream as the first image and is after the first image, and the parallax of the second image on the virtual screen is different from the parallax of the first image on the virtual screen when the second image is displayed.

[0064] Objects with large parallax appear closer, while objects with small parallax appear farther away. The stereoscopic effect when watching 3D videos mainly comes from the parallax generated by the left and right eyes. Different parallaxes will produce out-of-screen / in-screen effects. When the convergence point of 3D objects falls exactly on the display plane, it is called zero parallax.

[0065] When watching 3D videos, the greater the degree of in-screen and out-of-screen movement (i.e., the greater the absolute value of parallax), the stronger the 3D effect. However, if this exceeds a certain range (or exceeds a certain range for an extended period), it can lead to visual fatigue or difficulties in image fusion. In this embodiment, when displaying a 3D image, it is determined whether the parallax of the displayed image meets a preset condition. If the parallax meets the preset condition, the parallax of the displayed image is adjusted, and an image with a different parallax than the previously displayed image is displayed. For example, when the parallax of the first image on the virtual screen meets the preset condition, a second image with a different parallax than the first image can be rendered, thereby realizing the parallax adjustment of the image when displaying 3D videos.

[0066] In one possible implementation, the parallax on the virtual screen based on the first image satisfying a preset condition includes: the parallax on the virtual screen based on at least one display pixel of the first image satisfying a preset condition.

[0067] In one possible implementation, the condition that the parallax of the first image on the virtual screen meets a preset condition includes: the parallax of the first image on the virtual screen is not within the safe parallax range; and the parallax of the second image on the virtual screen is within the safe parallax range.

[0068] In one possible implementation, the 3D video frame stream can be traversed and the disparity value range of each frame pair (each pair of the same name, that is, the same point on the left and right eye images) can be calculated to determine whether it is within the safe disparity range. If the disparity on the virtual screen is not within the safe disparity range, the disparity of the displayed image can be adjusted.

[0069] Optionally, for images within the same video frame, the parallax of some images may have been adjusted, which can cause inter-frame jitter. Therefore, a motion averaging filter can be used to eliminate the large jitter caused by inter-frame parallax adjustment.

[0070] Reference Figure 4 , Figure 4 This is a schematic diagram of a specific process of the data processing method in the embodiments of this application.

[0071] In this embodiment of the application, when the parallax meets the preset conditions, it can be considered that the user's eyes are in a state of fatigue.

[0072] In one possible implementation, data about the user's eye state while viewing the first image can be acquired (e.g., through a sensor), and the data is related to the user's eye state; when the data indicates that the user's eyes are fatigued, the parallax of the first image on the virtual screen meets a preset condition.

[0073] In one possible implementation, the absolute value of the parallax of the second image on the virtual screen is less than the absolute value of the parallax of the first image on the virtual screen.

[0074] In one possible implementation, data related to the user's eye state can be acquired when the user is viewing the first image; when the data indicates that the user's eyes are in a healthy state, the parallax of the first image on the virtual screen meets a preset condition; the absolute value of the parallax of the second image on the virtual screen is greater than the absolute value of the parallax of the first image on the virtual screen.

[0075] There are significant individual differences in image fusion ability. People with strong image fusion ability can watch videos with a strong 3D feel, while in extreme cases, some people have such weak image fusion ability that they cannot watch 3D videos at all. This application can collect information about the user's eye state when watching 3D videos and adjust the parallax. When the eyes are fatigued, the absolute value of the parallax can be reduced to ensure the user's visual comfort. When the eyes are healthy, the absolute value of the parallax can be increased to enhance the 3D stereoscopic effect.

[0076] For example, the collected data may include interpupillary distance, eye movement correction parameters, three-dimensional eye movement coordinates, pupil size, blink signals, etc. Prolonged viewing of highly immersive 3D content can cause eye strain due to tension in the ciliary muscles or excessive adjustment of the binoculars' focus.

[0077] Reference Figure 5 , Figure 5 This is a schematic diagram of a specific process of the data processing method in the embodiments of this application.

[0078] In one possible implementation, this can be achieved by calculating the binocular convergence angle (e.g., Figure 7 The two main characteristics of visual fatigue determine whether to implement parallax accommodation and how to adjust it (e.g., ...). Figure 6 The adjustment flowchart can be as follows: Figure 8 As shown.

[0079] Specifically, fatigue-related eye movement features can be extracted in real time from time-series eye movement signals, including but not limited to pupil size, blink frequency, eyelid opening and closing degree, fixation duration, saccade distance, and gaze entropy. These features are then input into a time series model (such as HMM) and combined with historical sequence information to calculate the current fatigue level p(t).

[0080] Furthermore, the binocular convergence angle α can be calculated in real time using parameters such as eye-tracking three-dimensional coordinates and interpupillary distance (e.g., ...). Figure 9), calculate the cumulative convergence angle α within a certain time window, and the index f that describes the angle adjustment frequency (such as Figure 10 ).

[0081] When visual fatigue (p) exceeds the threshold, and the cumulative convergence angle (α) or accommodation frequency (f) is greater than the threshold, the parallax is reduced. When α and f are less than the threshold, no adjustment is made; when fatigue is less than the threshold, the parallax increases linearly over time until it reaches a safe threshold range or switches to other conditions. The adjustment range is determined by the stereoscopic level (low / medium / high) selected by the user before viewing. The adjustment window can be set to 0.2 seconds (the shortest time the human eye can distinguish changes in objects).

[0082] The cumulative convergence angle can be calculated based on the following formula:

[0083] The angle adjustment frequency can be calculated based on the following formula:

[0084] f = variance(a(t))

[0085] In one possible implementation, when displaying the first image according to the viewed parameters, an adjustment instruction for the parallax obtained from the user's input can be received, and it can be determined that the parallax of the first image on the virtual screen meets a preset condition.

[0086] In one possible implementation, the user input is at least one of the following: input via user gesture, input via keypad, or input via UI page.

[0087] In this embodiment, users can actively adjust the parallax according to their preferences or comfort through gestures / buttons / UI interaction.

[0088] When playing videos, the parallax can be manually adjusted according to the user's personal preference or comfort. This primarily includes, but is not limited to, the following interaction methods:

[0089] Method 1: Gesture control (e.g.) Figure 11 ): The screen entry / exit is determined by tracking the arm's pushing or pulling gestures. The parallax offset is linearly proportional to the pull-back / push-forward distance.

[0090] Method 2: Button control (e.g.) Figure 12 The direction of the button movement is used to determine whether the screen is in or out, and the parallax offset is linearly proportional to the number of button movements.

[0091] Method 3: UI control (e.g.) Figure 13 As shown in the figure, the parallax offset is linearly proportional to the slider drag distance.

[0092] This application enhances the interactivity and enjoyment of 3D video viewing by introducing parallax active adjustment methods such as gestures.

[0093] The following describes how to obtain the second image through rendering based on the first image and the viewing parameters:

[0094] In one possible implementation, a new viewing angle can be obtained based on the amount of parallax adjustment of the first image on the virtual screen; and a second image can be obtained by rendering based on the new viewing angle, the first image, and the viewing parameters.

[0095] In this embodiment, the target rendering viewpoint (i.e., the new viewing viewpoint) can be calculated based on the disparity offset, and a new viewpoint rendering algorithm (such as depth-based image-based rendering) can be applied. g The updated disparity map is obtained by IBR).

[0096] For example, a second image can be obtained by rendering based on the first image, its corresponding depth map, and the viewing parameters. By introducing a depth-based fast image rendering technique, information loss and distortion problems caused by cropping are avoided, and better parallax adjustment effects can be achieved.

[0097] This application embodiment can be applied to real-time adjustment of parallax in MR glasses for 3D photos or videos, maximizing the stereoscopic effect while ensuring comfort. The solution mainly includes two stages: video preprocessing before 3D video playback and online adjustment during playback. The preprocessing stage includes traversing all frames of the video stream and correcting frames that exceed the reasonable parallax range to within the reasonable range. The online parallax adjustment during playback includes two sub-modes: automatic parallax adjustment based on eye-tracking signal (physiological) feedback (mode one) and active parallax adjustment based on user preference / comfort (mode two). During online parallax adjustment, the user can choose to enable mode one or mode two, or both.

[0098] The methods provided in the embodiments of this application have been described in detail above. Next, the device for performing the above methods provided in the embodiments of this application will be described.

[0099] Please see Figure 14 , Figure 14 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. The apparatus includes:

[0100] The acquisition module 1401 is used to acquire a first image and viewing parameters, wherein the first image is a 3D image and the viewing parameters include information about the virtual screen;

[0101] For a detailed description of the acquisition module 1401, please refer to the description of step 301 in the above embodiment. The similarities will not be repeated here.

[0102] Processing module 1402 is configured to, when displaying the first image according to the viewing parameters, based on the parallax of the first image on the virtual screen satisfying a preset condition, render a second image according to the first image and the viewing parameters; the second image is an image belonging to the same 3D video stream as the first image and following the first image, and the parallax of the second image on the virtual screen when displaying the second image is different from the parallax of the first image on the virtual screen.

[0103] For a detailed description of the processing module 1402, please refer to the description of step 302 in the above embodiment. The similarities will not be repeated here.

[0104] In one possible implementation, the parallax on the virtual screen based on the first image satisfies a preset condition, including:

[0105] The parallax of at least one display pixel of the first image on the virtual screen satisfies a preset condition.

[0106] In one possible implementation, the parallax on the virtual screen based on the first image satisfies a preset condition, including:

[0107] Based on the fact that the parallax of the first image on the virtual screen is not within the safe parallax range;

[0108] The parallax of the second image on the virtual screen is within the safe parallax range.

[0109] In one possible implementation, the acquisition module 1401 is further configured to:

[0110] Data is acquired on the user's viewing of the first image, and the data is related to the user's eye condition;

[0111] When the data indicates that the user's eyes are fatigued, the parallax of the first image on the virtual screen meets a preset condition.

[0112] In one possible implementation, the absolute value of the parallax of the second image on the virtual screen is less than the absolute value of the parallax of the first image on the virtual screen.

[0113] In one possible implementation, the acquisition module 1401 is further configured to:

[0114] Data is acquired on the user's viewing of the first image, and the data is related to the user's eye condition;

[0115] When the data indicates that the user's eyes are in a healthy state, the parallax of the first image on the virtual screen meets a preset condition; the absolute value of the parallax of the second image on the virtual screen is greater than the absolute value of the parallax of the first image on the virtual screen.

[0116] In one possible implementation, the processing module 1402 is further configured to:

[0117] When displaying the first image based on the viewed parameters, an adjustment command for the parallax obtained based on the user's input is received, and it is determined that the parallax of the first image on the virtual screen meets a preset condition.

[0118] In one possible implementation, the user's input is at least one of the following:

[0119] Input can be made through user gestures, key presses, or UI page input.

[0120] In one possible implementation, the processing module 1402 is specifically used for:

[0121] A new viewing angle is obtained based on the amount of parallax adjustment of the first image on the virtual screen;

[0122] A second image is obtained by rendering based on the new viewing angle, the first image, and the viewing parameters.

[0123] In one possible implementation, obtaining a second image by rendering based on the first image and the viewing parameters includes: obtaining a second image by rendering based on the first image, the corresponding depth map, and the viewing parameters.

[0124] Please see Figure 15 , Figure 15 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 1500 can specifically be a server, a personal computer, a smartphone, etc., and is not limited thereto. Specifically, the execution device 1500 includes: a receiver 1501, a transmitter 1502, a processor 1503, and a memory 1504 (wherein the execution device 1500 may have one or more processors 1503). Figure 9 (Taking a processor as an example), processor 1503 may include application processor 15031 and communication processor 15032. In some embodiments of this application, receiver 1501, transmitter 1502, processor 1503 and memory 1504 may be connected via a bus or other means.

[0125] Memory 1504 may include read-only memory and random access memory, and provides instructions and data to processor 1503. A portion of memory 1504 may also include non-volatile random access memory (NVRAM). Memory 1504 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0126] Processor 1503 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses in the diagram are referred to as the bus system.

[0127] The methods disclosed in the embodiments of this application described above can be applied to processor 1503, or implemented by processor 1503. Processor 1503 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of processor 1503 or by instructions in the form of software. The processor 1503 described above can be a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0128] The processor 1503 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 1504. The processor 1503 reads information from memory 1504 and, in conjunction with its hardware, completes the steps of the above methods.

[0129] Receiver 1501 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1502 can be used to output digital or character information through the first interface; transmitter 1502 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1502 may also include a display device such as a display screen.

[0130] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0132] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0133] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, characterized in that, The method includes: Acquire a first image and viewing parameters, wherein the first image is a 3D image and the viewing parameters include information about the virtual screen; When the first image is displayed according to the viewing parameters, based on the parallax of the first image on the virtual screen satisfying a preset condition, a second image is obtained by rendering according to the first image and the viewing parameters; wherein, The second image is an image that belongs to the same 3D video stream as the first image and is placed after the first image. When the second image is displayed, the parallax on the virtual screen is different from that of the first image on the virtual screen.

2. The method according to claim 1, characterized in that, The condition that the parallax of the first image on the virtual screen meets a preset condition includes: The parallax of at least one display pixel of the first image on the virtual screen satisfies a preset condition.

3. The method according to claim 1 or 2, characterized in that, The condition that the parallax of the first image on the virtual screen meets a preset condition includes: Based on the fact that the parallax of the first image on the virtual screen is not within the safe parallax range; The parallax of the second image on the virtual screen is within the safe parallax range.

4. The method according to claim 1 or 2, characterized in that, The method further includes: Data is acquired on the user's viewing of the first image, and the data is related to the user's eye condition; When the data indicates that the user's eyes are fatigued, the parallax of the first image on the virtual screen meets a preset condition.

5. The method according to claim 4, characterized in that, The absolute value of the parallax of the second image on the virtual screen is less than the absolute value of the parallax of the first image on the virtual screen.

6. The method according to claim 1 or 2, characterized in that, The method further includes: Data is acquired on the user's viewing of the first image, and the data is related to the user's eye condition; When the data indicates that the user's eyes are in a healthy state, the parallax of the first image on the virtual screen meets a preset condition; The absolute value of the parallax of the second image on the virtual screen is greater than the absolute value of the parallax of the first image on the virtual screen.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: When displaying the first image based on the viewed parameters, an adjustment command for the parallax obtained based on the user's input is received, and it is determined that the parallax of the first image on the virtual screen meets a preset condition.

8. The method according to claim 7, characterized in that, The user's input is at least one of the following: Input can be made through user gestures, key presses, or UI page input.

9. The method according to any one of claims 1 to 8, characterized in that, The step of rendering a second image based on the first image and the viewing parameters includes: A new viewing angle is obtained based on the amount of parallax adjustment of the first image on the virtual screen; A second image is obtained by rendering based on the new viewing angle, the first image, and the viewing parameters.

10. The method according to any one of claims 1 to 9, characterized in that, The step of rendering a second image based on the first image and the viewing parameters includes: A second image is obtained by rendering based on the first image, the corresponding depth map, and the viewing parameters.

11. A data processing apparatus, characterized in that, The device includes: An acquisition module is used to acquire a first image and viewing parameters, wherein the first image is a 3D image and the viewing parameters include information about the virtual screen. The processing module is configured to, when displaying the first image according to the viewing parameters, render a second image based on the parallax of the first image on the virtual screen satisfying a preset condition, according to the first image and the viewing parameters; the second image is an image belonging to the same 3D video stream as the first image and following the first image, and the parallax of the second image on the virtual screen when displaying the second image is different from the parallax of the first image on the virtual screen.

12. The apparatus according to claim 11, characterized in that, The condition that the parallax of the first image on the virtual screen meets a preset condition includes: The parallax of at least one display pixel of the first image on the virtual screen satisfies a preset condition.

13. The apparatus according to claim 11 or 12, characterized in that, The condition that the parallax of the first image on the virtual screen meets a preset condition includes: Based on the fact that the parallax of the first image on the virtual screen is not within the safe parallax range; The parallax of the second image on the virtual screen is within the safe parallax range.

14. The apparatus according to claim 11 or 12, characterized in that, The acquisition module is also used for: Data is acquired on the user's viewing of the first image, and the data is related to the user's eye condition; When the data indicates that the user's eyes are fatigued, the parallax of the first image on the virtual screen meets a preset condition.

15. The apparatus according to claim 14, characterized in that, The absolute value of the parallax of the second image on the virtual screen is less than the absolute value of the parallax of the first image on the virtual screen.

16. The apparatus according to claim 11 or 12, characterized in that, The acquisition module is also used for: Data is acquired on the user's viewing of the first image, and the data is related to the user's eye condition; When the data indicates that the user's eyes are in a healthy state, the parallax of the first image on the virtual screen meets a preset condition; The absolute value of the parallax of the second image on the virtual screen is greater than the absolute value of the parallax of the first image on the virtual screen.

17. The apparatus according to any one of claims 11 to 16, characterized in that, The processing module is further configured to: When displaying the first image based on the viewed parameters, an adjustment command for the parallax obtained based on the user's input is received, and it is determined that the parallax of the first image on the virtual screen meets a preset condition.

18. The apparatus according to claim 17, characterized in that, The user's input is at least one of the following: Input can be made through user gestures, key presses, or UI page input.

19. The apparatus according to any one of claims 11 to 18, characterized in that, The processing module is specifically used for: A new viewing angle is obtained based on the amount of parallax adjustment of the first image on the virtual screen; A second image is obtained by rendering based on the new viewing angle, the first image, and the viewing parameters.

20. The apparatus according to any one of claims 11 to 19, characterized in that, The processing module is specifically used to: obtain a second image by rendering based on the first image, the corresponding depth map, and the viewing parameters.

21. A terminal device, characterized in that, The terminal device includes a processor, a memory, a display module, and a bus, wherein: The processor, the display screen, and the memory are connected via the bus; The memory is used to store computer programs; The processor is configured to control the memory, execute the program stored in the memory, and control the display module to implement the steps of the method according to any one of claims 1 to 10.

22. A computer-readable storage medium comprising a program, which, when executed on a computer, causes the computer to perform the method as claimed in any one of claims 1 to 10.

23. A computer program product containing instructions, characterized in that, When the computer program product is run on a terminal, the terminal causes the terminal to perform the method described in any one of claims 1-10.

Images