Image display method, device, apparatus, and storage medium

Abstract

The application discloses an image display method, device, equipment and storage medium. The method comprises the following steps: in response to a virtual-real switching event that a user switches from a virtual environment to a real environment, acquiring a current video stream, camera pose data corresponding to each image frame in the current video stream, and viewpoint pose data of the user at a current time, the current video stream being obtained by shooting a real environment in which the user is located through a camera on a head-mounted display device worn by the user; training an image rendering model based on the current video stream and the camera pose data corresponding to each image frame; performing image rendering processing based on the viewpoint pose data at the current time through the trained image rendering model to obtain a target rendering image that is adapted to the viewpoint of the user; and displaying the target rendering image through the head-mounted display device, the target rendering image being used to reflect the real environment observed from the viewpoint of the user, so that the user can perform activities in the real environment based on the target rendering image.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to an image display method, apparatus, device, and storage medium. Background Technology

[0002] As a core subfield of the metaverse, virtual reality (VR) can provide entirely different digital virtual environments outside of people's daily real-life environments through computer technology. These virtual environments are diverse, ranging from game environments created free from the constraints of the real world to highly realistic simulations of the real environment. Under current mainstream technology, users enter the virtual environment through head-mounted display (HMD) devices. HMDs, by obscuring the user's eyes and displaying images of the virtual environment, immerse the user's perception in the virtual world, shifting their focus from the real world.

[0003] However, while users are visually immersed in the virtual environment, their bodies remain in the real environment. This fact necessitates that HMDs completely block users' perception of the real environment in order to enhance the immersive experience. That is, even when users are working in a virtual environment, many things still occur in the real environment; therefore, sometimes it is necessary to maintain some awareness of the real environment. In such cases, if users frequently switch between the real and virtual environments by removing and putting on the HMD, it will severely damage the user experience.

[0004] Therefore, how to switch the user's field of vision to the real environment without requiring the user to remove the HMD is a problem that urgently needs to be solved. Summary of the Invention

[0005] The purpose of this application is to provide an image display method, apparatus, device, and storage medium that can switch the user's field of view to the real environment without requiring the user to remove the HMD.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide an image display method, including:

[0008] In response to a virtual-to-real switching event where the user switches from a virtual environment to a real environment, the system acquires the current video stream, camera pose data corresponding to each image frame in the current video stream, and the user's viewpoint pose data at the current moment. The current video stream is obtained by capturing the real environment in which the user is located through a camera on the head-mounted display device worn by the user, and the current video stream includes multiple image frames arranged in a time sequence.

[0009] The image rendering model is trained based on the current video stream and the camera pose data corresponding to each image frame in the current video stream.

[0010] The trained image rendering model performs image rendering processing based on the viewpoint pose data at the current moment to obtain a target rendered image adapted to the user's viewpoint.

[0011] The target rendered image is displayed through the head-mounted display device, wherein the target rendered image is used to reflect the real environment as observed from the user's perspective, so that the user can perform activities in the real environment based on the target rendered image.

[0012] Secondly, embodiments of this application provide an image display device, comprising:

[0013] The acquisition unit is used to acquire the current video stream, the camera pose data corresponding to each image frame in the current video stream, and the user's viewpoint pose data at the current moment in response to a virtual-real switching event where the user switches from a virtual environment to a real environment. The current video stream is obtained by capturing the real environment in which the user is located through a camera on the head-mounted display device worn by the user, and the current video stream includes multiple image frames arranged in a time sequence.

[0014] The training unit is used to train the image rendering model based on the current video stream and the camera pose data corresponding to each image frame in the current video stream;

[0015] The rendering unit is used to perform image rendering processing based on the viewpoint pose data at the current moment using a trained image rendering model to obtain a target rendered image adapted to the user's viewpoint.

[0016] The display unit is used to display the target rendered image through the head-mounted display device, wherein the target rendered image is used to reflect the real environment observed from the user's perspective, so that the user can perform activities in the real environment based on the target rendered image.

[0017] Thirdly, embodiments of this application provide an image display device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method as described in the first aspect.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium that, when instructions in the storage medium are executed by a processor of an image display device, enables the image display device to perform the method described in the first aspect.

[0019] The at least one technical solution adopted in this application embodiment can achieve the following beneficial effects: A video stream of the user's real environment is captured by a camera on a user-worn head-mounted display device; then, an image rendering model is trained based on the current video stream and the camera pose data corresponding to each image frame in the current video stream, enabling the trained image rendering model to render images adapted to the camera pose data; furthermore, since camera pose data can describe the position of the camera when capturing an image, similarly, the user's viewpoint pose data can describe the position of the user's eyes when viewing an image. The trained image rendering model then renders the image based on the user's viewpoint pose data at the current moment. Like rendering, a target rendered image adapted to the user's viewpoint can be obtained and displayed through a head-mounted display device. This allows the user to perceive the real environment without removing the head-mounted display device, and because the target rendered image is adapted to the user's viewpoint, the spatial relationships of objects in the real environment perceived by the user through the head-mounted display device are accurate. This enables the user to accurately interact with or manipulate objects in the real environment based on the target rendered image displayed by the head-mounted display device, such as locating and grasping objects in the real environment, which is beneficial to improving the user experience. Furthermore, since the objects needed by users in the real environment are accidental, and the target rendered image displayed by the head-mounted display device in this embodiment is obtained by training an image rendering model based on the video stream captured by the camera in the real environment where the user is located and the camera posture data, the image rendering model is then used to render the image based on the viewpoint posture data at the current moment. Therefore, the image display method provided in this embodiment can be applied to the user's accidental interaction needs. That is, the user does not need to foresee which objects in the real environment he or she will interact with before entering the virtual environment, nor does he or she need to know the location of the objects to be interacted with in advance. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a schematic diagram illustrating an application scenario of an image display method provided in one embodiment of this application.

[0022] Figure 2 A schematic flowchart illustrating an image display method according to an embodiment of this application;

[0023] Figure 3 A schematic diagram of a user viewpoint and a camera viewpoint provided for one embodiment of this application;

[0024] Figure 4 A schematic diagram illustrating the motion trajectory and coverage area of ​​a camera on a head-mounted display device according to an embodiment of this application;

[0025] Figure 5 A flowchart illustrating an image display method according to another embodiment of this application;

[0026] Figure 6 A schematic diagram of the structure of an image display device provided in one embodiment of this application;

[0027] Figure 7 This is a schematic diagram of the structure of an image display device provided in one embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] The terms "first," "second," etc., used in this specification and claims are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in this specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0030] Explanation of some concepts:

[0031] Virtual Reality (VR) is a computer simulation system that can create and experience virtual environments. It uses computers to generate a simulated environment that allows users to immerse themselves in it.

[0032] Head-mounted display (HMD): Also known as VR glasses, VR headsets, etc. It is a device used to present virtual scenes to a viewer. A typical HMD completely covers the wearer's eyes, completely blocking out the real environment, thus providing an immersive experience. Typically, an HMD has two internal displays, providing images to the left and right eyes respectively. Specifically, to provide a 3D viewing experience, the images displayed to the left and right eyes are not exactly the same, but rather a pair of stereoscopic images created based on the difference in viewpoints between the two eyes.

[0033] Viewpoint: A virtual concept used to describe the camera's position when an image is captured. Similarly, it can describe the position of the human eye when viewing an image. Specifically, the viewpoint of the left eye differs from that of the right eye. It is precisely because of this difference in viewpoints that the images seen by the left and right eyes are not entirely identical. For example, if you hold a finger in front of your nose, blocking your left and right eyes in turn, you will see that the objects appear to move laterally; this phenomenon is called parallax. Parallax is the primary basis for humans to perceive the 3D position of objects (but not the only one; a single eye can also allow a person to locate and grasp an object).

[0034] Purely observed objects: Objects in the real environment that do not require user interaction, such as paintings on the wall.

[0035] Interactive objects: Objects in the real environment that need to be interacted with by the user, such as a water cup.

[0036] The core difference between these two types of objects lies in the fact that, for objects to be interacted with, users in a virtual environment need to interact with or manipulate objects in the real environment, thus requiring accurate spatial positioning of the real objects. However, simply observing an object only requires a video stream image of the real environment and does not require determining its spatial positioning.

[0037] To switch a user's field of vision to the real environment without requiring the user to remove their head-mounted display (HMD), this application provides an image display method. This method involves capturing a video stream of the user's real-world environment using a camera on the HMD. Then, an image rendering model is trained based on the current video stream and the camera pose data corresponding to each image frame within that stream. This trained model renders images adapted to the camera pose data. Furthermore, since camera pose data describes the camera's position when capturing an image, similarly, the user's viewpoint pose data describes the position of the user's eyes when viewing the image. The trained image rendering model, based on the user's viewpoint pose data, can then render images that adapt to the camera pose data. By rendering images based on the current viewpoint posture data, a target rendering image adapted to the user's viewpoint can be obtained. This target rendering image is then displayed through a head-mounted display device, allowing the user to perceive the real environment without removing the device. Furthermore, because the target rendering image is adapted to the user's viewpoint, the spatial relationships of objects in the real environment perceived by the user through the head-mounted display device are accurate. This enables the user to accurately interact with or manipulate objects in the real environment based on the target rendering image displayed on the head-mounted display device, such as locating and grasping objects, thus improving the user experience.

[0038] It should be understood that the image display method proposed in this application embodiment can be executed by an image display device or software installed in the image display device. The image display device can be a head-mounted display device, or other devices connected to the head-mounted display device. For example, the image display device includes a terminal or server other than a head-mounted display. A terminal can include a laptop computer, tablet computer, vehicle-mounted terminal, smart interactive device, etc., and a server can include an independent physical server, a server cluster consisting of one or more physical servers, or a cloud server capable of cloud computing.

[0039] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0040] To enable those skilled in the art to understand the technical solutions provided in the embodiments of this application, the following is combined with... Figure 1 The illustration depicts a practical application scenario, providing a detailed description of the technical solutions provided in the embodiments of this application. It should be understood that the technical solutions provided in the embodiments of this application are applied to... Figure 1The scenario shown is merely an illustrative example and should not be construed as limiting the application scenario of the embodiments of this application. This implementation scenario includes an image display device, which may be a head-mounted display device 1, or other devices connected to the head-mounted display device (not shown in the figure), etc.

[0041] like Figure 1 As shown, the user enters the virtual environment through a head-mounted display device 1. The display device 1 is equipped with at least one camera 11. Figure 1 The illustration uses two cameras. Camera 11 can be used to capture images of the user's real-world environment. In practical applications, various types of cameras can be used, such as RGB-D cameras, RGB cameras, etc., and this embodiment does not limit the specific type. Preferably, considering that RGB-D cameras capture images based on active detection, their performance is somewhat compromised in sunlight, and they also have poor image quality for highly reflective objects (such as ceramic cups). Therefore, an RGB camera can be used, thus broadening its applicability.

[0042] When a user needs to perceive the real environment 2, the head-mounted display device 1 can be triggered to activate the virtual-to-real switching function, switching the user's field of vision from the virtual environment to the real environment. Upon receiving the user's trigger command, the head-mounted display device 1 captures a video stream of the user's environment through the camera 11. Based on the video stream, the camera pose data corresponding to each image frame in the video stream, and the user's viewpoint pose data, it performs image rendering processing to display a target rendered image adapted to the user's viewpoint. In this way, the user does not need to remove the head-mounted display device but can directly perceive the real environment through the target rendered image displayed on the head-mounted display device. Furthermore, because the target rendered image is adapted to the user's viewpoint, the spatial relationships of objects in the real environment perceived by the user through the head-mounted display device are accurate. This allows the user to accurately interact with or manipulate objects in the real environment based on the target rendered image displayed on the head-mounted display device, such as locating and grasping objects in the real environment, thus improving the user experience.

[0043] Optionally, the head-mounted display device 1 can also be wirelessly connected to other terminals, which can then execute the image display method described above.

[0044] The technical solutions provided in this application will describe in detail the specific implementation process of image rendering and display by an image display device. The technical solutions provided in each embodiment of this application will be described in detail below with reference to the accompanying drawings.

[0045] Please see Figure 2 The following is a flowchart illustrating an image display method according to an embodiment of this application. The method may include the following steps:

[0046] S202, in response to a virtual switching event where the user switches from a virtual environment to a real environment, acquires the current video stream, the camera pose data corresponding to each image frame in the current video stream, and the user's viewpoint pose data at the current moment.

[0047] The current video stream refers to the video within a preset time period prior to the current moment. The current video stream is captured by a camera on a user-worn head-mounted display device, showing the user's real-world environment. The current video stream includes multiple image frames arranged in chronological order. The length of the preset time period can be selected according to actual needs; this embodiment does not limit this selection.

[0048] In this embodiment, the camera on the head-mounted display device captures the user's real-time environment. "Real-time" here means a short interval between two captures, such as 1 second or 2 seconds. The specific interval can be set according to actual needs, and this embodiment does not limit this. In other words, the current video stream includes image frames from multiple moments, including the current moment and multiple historical moments preceding it.

[0049] Because the camera on a head-mounted display changes with the user's head movements, the camera's viewpoint may be different at different times. Each image frame in the current video stream is captured by the camera at different times, therefore each image frame has corresponding camera pose data. The camera pose data corresponding to each image frame represents the camera's viewpoint pose at the time the image frame was captured, specifically including the camera's position and angle in the global coordinate system.

[0050] In S202 above, the camera pose data corresponding to the image frame can be obtained through various appropriate methods, which can be selected according to actual needs. This application embodiment does not limit this. Optionally, in order to ensure the accuracy of the camera pose data, the camera pose data corresponding to each image frame in the current video stream can be obtained in the following way: based on a preset pose tracking algorithm, multiple image frames in the current video stream are tracked and pose estimated to obtain the camera pose data corresponding to each image frame in the current video stream.

[0051] The preset attitude tracking algorithm can be any of the attitude tracking algorithms commonly used in the field, and can be set according to actual needs. This application embodiment does not limit this.

[0052] For example, the preset attitude tracking algorithm can be the ORB-SLAM3 algorithm. In this case, the ORB algorithm can be used to extract features from each image frame in the current video stream to obtain the ORB features of each image frame; then, IMU pre-integration is performed based on the ORB features of the previous image frame to deduce the camera attitude data corresponding to the next image frame.

[0053] The user's viewpoint pose data at the current moment is used to represent the user's viewpoint pose at the current moment. For example... Figure 3 As shown, the user's viewpoint includes a left-eye viewpoint and a right-eye viewpoint. Correspondingly, the user's viewpoint pose data can include left-eye pose data and right-eye pose data. The left-eye pose data represents the position and angle of the user's left-eye viewpoint in the global coordinate system, and the right-eye pose data represents the position and angle of the user's right-eye viewpoint in the global coordinate system. For example,

[0054] To ensure that the acquired viewpoint pose data accurately describes the user's eye pose, thereby ensuring that the subsequently rendered image adapts to the user's viewpoint, the user's viewpoint pose data at the current moment can be obtained as follows: The user's relative viewpoint pose data is obtained, whereby the relative viewpoint pose data represents the user's viewpoint pose relative to the camera; further, based on the product of the relative viewpoint pose data and the camera pose data corresponding to the current image frame in the current video stream, the user's viewpoint pose data at the current moment is determined. In practical applications, considering the parallax between the user's left and right eyes, to further ensure that the subsequently rendered image adapts to the user's viewpoint, the user's relative viewpoint pose data includes the relative viewpoint pose data for the left eye and the right eye. The left eye relative viewpoint pose data represents the pose of the user's left eye viewpoint relative to the camera, and the right eye relative viewpoint pose data represents the pose of the user's right eye viewpoint relative to the camera.

[0055] Specifically, since the head-mounted display device is a rigid body, the user's viewpoint relative attitude data can be pre-calibrated according to the size of the head-mounted display device. The specific calibration process can be implemented using various calibration methods commonly used in this field, which will not be elaborated further. Alternatively, to improve the accuracy of the viewpoint relative attitude data, it can also be calculated in real time using the camera built into the head-mounted display device used to observe the user's eyes. The specific calculation process can be implemented using various attitude calculation methods commonly used in this field, which will not be elaborated further.

[0056] Assume the relative pose data of the left eye's viewpoint is denoted as... The relative pose data of the right eye's viewpoint is denoted as The camera pose data corresponding to the image frame at the current moment is Therefore, the user's left eye pose data at the current moment can be obtained as follows: The user's right eye pose data at the current moment is:

[0057] In practical applications, the head-mounted display device can activate the virtual-to-real switching function at any appropriate time, and the specific settings can be configured according to actual needs. This application embodiment does not limit this. Optionally, in the above S202: during the process of displaying virtual environment images through the head-mounted display device, it is detected whether a virtual-to-real switching event is triggered. If so, the above-mentioned acquisition of the current video stream, the camera posture data corresponding to each image frame in the current video stream, and the user's viewpoint posture data at the current moment are executed. Among them, the virtual-to-real switching event includes at least one of the following events: the user inputs a voice command to switch the virtual reality scene to the real environment, the user triggers the virtual-to-real switching button on the head-mounted display device, the user performs a preset tapping operation on the head-mounted display device, the user's head moves according to a preset movement mode, etc.

[0058] For example, a head-mounted display device may have voice control functionality. If the head-mounted display device receives the user's voice input of "switch to real environment," it determines that a virtual-to-real transition event has been triggered. Alternatively, the head-mounted display device may be equipped with a virtual-to-real transition button. If the head-mounted display device detects the user's activation of the virtual-to-real transition button while displaying a virtual environment image, it determines that a virtual-to-real transition event has been triggered. Similarly, if the user's activation of the virtual-to-real transition button is detected again, the device switches back from the real environment to the virtual environment and continues displaying the virtual environment image. Furthermore, the head-mounted display device may also be equipped with a touchpad. If the head-mounted display device detects the user's head moving according to a preset motion, such as nodding or continuous head shaking, it determines that a virtual-to-real transition event has been triggered, and so on.

[0059] Understandably, when a head-mounted display device detects a virtual-to-real transition event, it activates the function to switch to the real environment and executes the aforementioned S202. This not only meets the user's actual virtual-to-real transition needs and improves the user experience, but also saves processing resources. Furthermore, by configuring various virtual-to-real transition events, users can conveniently and quickly switch between virtual and real environments, further enhancing the user experience.

[0060] S204. The image rendering model is trained based on the current video stream and the camera pose data corresponding to each image frame in the current video stream.

[0061] In this context, the image rendering model refers to a model with image rendering capabilities. In this embodiment, the image rendering model can have any suitable structure, and the specific choice can be made according to actual needs; this embodiment does not limit this. Optionally, to ensure image rendering effects, the image rendering model can employ a Neural Rendering (NeRF) model. The NeRF model generates images by tracing light rays entering the scene and integrating the light ray length. Specifically, it can be based on the camera pose data corresponding to each image frame in the video stream to track the light rays entering the scene and integrate their length, thereby obtaining the corresponding rendered image.

[0062] In this embodiment, the image rendering model can be trained in any appropriate way, and the specific training method can be selected according to actual needs. This embodiment does not limit this method. Image rendering models are generally designed for static environments, but this embodiment also considers the potential interaction between the user and objects in their real-world environment. Therefore, this embodiment is designed for dynamic environments. Extensive research has shown that, under normal circumstances, user head movements are relatively gentle, i.e., there are no rapid, large-angle head turns, and the camera on the head-mounted display device also follows the user's head movements relatively gently. Within a short timeframe (e.g., 500ms), the real environment can be approximated as static, and multiple image frames within this short timeframe can be considered as images captured by the camera from different angles of the user's real-world environment.

[0063] For example, Figure 4 This diagram illustrates the motion trajectory and coverage area of ​​a camera on a head-mounted display device. Figure 4 In the diagram, black dots represent the camera's viewpoint, dashed lines represent the camera's movement trajectory, and triangles represent the camera's coverage area. Within a short timeframe, based on the camera's movement trajectory, it's easy to see that the image frames captured by the camera at various moments within that timeframe can be considered as images of the user's real environment captured by the camera from different angles.

[0064] For example, taking a window length of 5, meaning a short time period contains 5 moments, then for the current moment t4, its corresponding time window is T0 = [t0, t1, t2, t3, t4], and the current video stream is... in, Let be the image frame captured by the k-th camera at time t. Then, for the new current time t5, its corresponding time window is T1 = [t1, t2, t3, t4, t5], and the current video stream is... As can be seen, the multiple image frames contained in the current video stream can be regarded as images captured by each camera on the head-mounted display device from different angles of the user's real environment.

[0065] Therefore, a sliding window can be used to apply time dimension to the current video stream to obtain multiple image frames within a short time period. Then, the image rendering model can be trained based on the multiple image frames and the camera pose data corresponding to each of these multiple image frames. The resulting image rendering model is suitable for image rendering in dynamic real-world environments.

[0066] Specifically, in one alternative implementation, S204 may include the following steps:

[0067] S241, Based on multiple image frames in the first time period of the current video stream and the camera pose data corresponding to each image frame, train the image rendering model to obtain the first image rendering model.

[0068] The first time period can be a relatively short time period preceding the current time, such as the time period T0 = [t3, t4, t5, t6, t7] which includes the current time and the four historical times preceding it. Multiple image frames within the first time period are... in, Let be the image frame captured by the k-th camera at time t. Correspondingly, the camera pose data corresponding to multiple image frames within the first time interval are: in, This represents the pose data of the viewpoint of the k-th camera at time t.

[0069] Specifically, in S241 above, the image rendering model can perform image rendering processing based on the camera pose data corresponding to each image frame in the first time period to obtain the rendered image corresponding to each image frame in the first time period; then, based on the difference information between each image frame in the first time period and the rendered image corresponding to each image frame, the model parameters of the image rendering model are adjusted to obtain the first image rendering model.

[0070] The model parameters of the image rendering model may include, but are not limited to, the number of processing nodes (such as neurons) in the image rendering model, the connection relationships between processing nodes in different network layers and the weights of the connection edges, and the biases corresponding to the processing nodes in each network layer.

[0071] In step S241 above, the rendering loss of the image rendering model can be determined based on a preset loss function, each image frame within the first time period, and the rendered image corresponding to each image frame. Further, the model parameters of the sub-image rendering model are adjusted based on the gradient descent algorithm, the backpropagation algorithm, and the rendering loss of the image rendering model. This process is repeated multiple times until a preset training stopping condition is met, thereby obtaining the first image rendering model. The rendering loss of the image rendering model represents the difference information between the rendered image corresponding to each image frame and each image frame, thus reflecting the learning effect of the image rendering model and ensuring that the rendered image obtained by the trained first image rendering model accurately reflects the user's real environment. In practical applications, any appropriate loss function can be used for the preset loss function, and the specific function can be selected according to actual needs; this application embodiment does not limit this. The preset training stopping condition can also be set according to actual needs, such as the rendering loss of the image rendering model being less than a preset loss threshold or the number of iterations reaching a preset number threshold; this application embodiment does not limit this.

[0072] S242, based on multiple image frames in the second time period of the current video stream and the camera pose data corresponding to each image frame, the first image rendering model is trained to obtain the second image rendering model.

[0073] The second time period can be a time period preceding the current moment that is longer than the first time period. Specifically, the second time period includes the first time period. For example, the time period T1 = [t0, t1, t2, t3, t4, t5, t6, t7] includes the current moment and the seven historical moments preceding it. The multiple image frames within the first time period are... in, Let be the image frame captured by the k-th camera at time t. Correspondingly, the camera pose data corresponding to multiple image frames within the second time period are: in, This represents the pose data of the viewpoint of the k-th camera at time t.

[0074] To eliminate camera pose data The cumulative error in the process can be corrected in S242 by using a preset attitude tracking algorithm and multiple image frames in the second time period to obtain corrected camera attitude data for each image frame in the second time period. Furthermore, the first image rendering model is trained based on multiple image frames in the second time period and the corrected camera attitude data for each image frame to obtain the second image rendering model.

[0075] For example, the preset attitude tracking algorithm can be the ORB-SLAM3 algorithm. In this case, the bundle adjustment (BA) method in the ORB-SLAM3 algorithm can be used, combined with the point cloud map at each time point in the second time period, to correct the camera attitude data HP corresponding to each image frame in the second time period, thus obtaining the corrected camera attitude data HP′ corresponding to each image frame in the second time period. The specific implementation process of attitude data correction using the ORB-SLAM3 algorithm can be implemented using various attitude correction methods commonly used in this field, and will not be elaborated further.

[0076] Understandably, the acquisition of camera pose data in S202 primarily considers time efficiency, thus employing a tracking method. This method calculates the camera pose at the current moment based on local information. While fast, this tracking method suffers from cumulative errors. For example, estimations based on information from time T at time T+1 may produce some errors, and estimations based on information from time T+2 at time T+1 may produce new errors. Furthermore, old errors continue to propagate, leading to significant error accumulation over time. Therefore, it is necessary to use relatively global information for overall adjustment and error elimination. Based on this, before training the first image rendering model, the camera pose data corresponding to each image frame within the second time period is corrected based on a preset pose tracking algorithm and multiple image frames within the second time period. Then, the first image rendering model is trained based on these multiple image frames and the corrected camera pose data. This reduces the estimation error of the camera pose data, thereby improving the rendering effect of the image rendering model.

[0077] It is worth noting that the specific implementation process of training the first image rendering model based on multiple image frames in the second time period and the corrected camera pose data corresponding to each image frame is similar to S241 above. Please refer to the detailed description of S241 above, and it will not be repeated here.

[0078] It is understandable that training the image rendering model based on multiple image frames within the first time period and the camera pose data corresponding to each image frame can achieve a local optimization effect on the image rendering model. Furthermore, training the first image rendering model based on multiple image frames within the second time period in the current video stream and the camera pose data corresponding to each image frame is equivalent to performing global optimization on the locally optimized image rendering model, which can further improve the rendering effect of the image rendering model and ensure that the image rendered by the image rendering model can be adapted to the user's viewpoint.

[0079] In practical applications, in order to balance the contradiction between rendering accuracy and real-time performance, a dual-thread mechanism can be adopted. The time-consuming but effective global optimization process is placed in the background thread for low-frequency execution, while the fast-converging and low-time local optimization process is placed in the foreground thread. This allows the foreground thread to ensure rendering real-time performance, alleviate the problem of a small number of image frames in the first time period, and stabilize the local optimization of the image rendering model.

[0080] Specifically, in S241 above, the image rendering model can be trained by the foreground thread based on multiple image frames within a first time period and the camera pose data corresponding to each image frame to obtain a first image rendering model. In S242 above, the first image rendering model can be trained by the background thread based on multiple image frames within a second time period and the camera pose data corresponding to each image frame to obtain a second image rendering model.

[0081] This application embodiment illustrates one specific implementation of the above-described S204. Of course, it should be understood that S204 can also be implemented in other ways, and this application embodiment does not limit this implementation. For example, in the above-described S204, only S241 can be executed, that is, based on multiple image frames within a first time period and the camera pose data corresponding to each image frame, the image rendering model is trained, and the trained first image rendering model is used for subsequent image rendering.

[0082] S206 uses the trained image rendering model to perform image rendering processing based on the viewpoint pose data at the current moment, and obtains a target rendered image that is adapted to the user's viewpoint.

[0083] Specifically, when optimizing the image rendering model using dual threads, if the training of the first image rendering model in the background thread does not meet the preset training stop condition, then the first image rendering model is used in the foreground thread to perform image rendering processing based on the viewpoint pose data at the current moment, resulting in a target rendered image adapted to the user's viewpoint; or, if the training of the first image rendering model in the background thread meets the preset training stop condition, then the first image rendering model that meets the preset training stop condition is used as the second image rendering model, and the second image rendering model is used in the foreground thread to perform image rendering processing based on the viewpoint pose data at the current moment, resulting in a target rendered image adapted to the user's viewpoint.

[0084] By performing local optimization on the image rendering model, the first image rendering model obtained after local optimization can be used to perform image rendering processing based on the viewpoint pose data at the current moment, so as to obtain a target rendered image that is adapted to the user's viewpoint.

[0085] In the process of image rendering based on the current world viewpoint pose data, the trained image rendering model can combine the user's current left-eye viewpoint pose data and the field of view (FOV) of the head-mounted display device, and use the MobileNeRF rendering method to process the image, obtaining a target rendered image adapted to the left-eye viewpoint. Similarly, the trained image rendering model can also combine the user's current right-eye viewpoint pose data and the FOV of the head-mounted display device, and use the MobileNeRF rendering method to process the image, obtaining a target rendered image adapted to the right-eye viewpoint. The FOV of the head-mounted display device is pre-calibrated or obtained according to factory configuration.

[0086] Optionally, in order to save computing resources and reduce image display latency, before S206 above, the image display method provided in this application embodiment may further include:

[0087] S208 displays the target rendered image via a head-mounted display device.

[0088] The target rendered image is used to reflect the real environment as observed from the user's perspective, so that the user can perform activities in the real environment based on the target rendered image. These activities can include interacting with objects in the real environment, moving in the real environment, and observing the real environment.

[0089] Specifically, a target rendering image adapted to the user's left eye and a target rendering image adapted to the user's right eye can be displayed through a head-mounted display device.

[0090] Optionally, to conserve computing resources and reduce display latency, before step S204 above, the image display method provided in this embodiment may further include: performing image recognition on the image frame at the current moment to determine whether there is an interactive object in the user's real environment at the current moment. The object in the real environment that needs to be interacted with by the user is, for example, a water cup. In practical applications, various image recognition technologies commonly used in the field can be employed to perform image recognition on the image frame at the current moment, which will not be elaborated further.

[0091] Accordingly, S204 may include: if there is an object to be interacted with in the user's real environment, then the image rendering model is trained based on the current video stream and the camera pose data corresponding to each image frame in the current video stream. In other words, if there is an object to be interacted with in the user's real environment, then S204 to S208 are executed.

[0092] Understandably, when users interact with objects in their real-world environment, accurate spatial positioning of the objects is required. The presence of interactive objects in the user's real-world environment, along with image rendering and display using a trained image rendering model, ensures that the rendered image is adapted to the user's viewpoint. This allows the user to perceive the correct spatial relationships of objects in the real-world environment through the head-mounted display device. Consequently, the user can accurately interact with or manipulate objects in the real-world environment based on the target image displayed on the head-mounted display device, such as locating and grasping objects, thus improving the user experience.

[0093] Optionally, after performing image recognition on the image frame at the current moment to determine whether there is an object to be interacted with in the real environment where the user is located at the current moment, the image display method provided in this application embodiment may further include: if there is no object to be interacted with in the real environment where the user is located, then displaying the image frame at the current moment through a head-mounted display device.

[0094] Understandably, when there are no objects to interact with in the user's real-world environment, there is no need for accurate spatial positioning of objects in the real environment. In this case, directly displaying the current image frame through a head-mounted display device can eliminate the need for training and rendering the image rendering model, thereby saving computing resources and reducing display latency.

[0095] Please see Figure 5 The following is a flowchart illustrating an image display method according to another embodiment of this application. The method may include the following steps:

[0096] S502 displays virtual environment images via a head-mounted display device.

[0097] S504, detects whether a virtual-to-real switching event has been triggered.

[0098] If yes, proceed with steps S506 to S508; otherwise, continue with step S502.

[0099] S506, acquire the current video stream, the camera pose data corresponding to each image frame in the current video stream, and the user's viewpoint pose data at the current moment.

[0100] S508 performs image recognition on the image frame at the current moment to determine whether there is an object to be interacted with in the real environment where the user is currently located.

[0101] If not, execute S510; if yes, execute S512 to S516.

[0102] The S510 displays an image frame of the current moment via a head-mounted display device.

[0103] S512 trains the image rendering model based on the current video stream and the camera pose data corresponding to each image frame in the current video stream.

[0104] S514 uses the trained image rendering model to perform image rendering processing based on the viewpoint pose data at the current moment, and obtains a target rendered image that is adapted to the user's viewpoint.

[0105] S516 displays a target rendered image via a head-mounted display device.

[0106] The target rendered image is used to reflect the real environment as observed from the user's perspective, so that the user can perform activities in the real environment based on the target rendered image. These activities can include interacting with objects in the real environment, moving in the real environment, and observing the real environment.

[0107] One or more embodiments of this application provide an image display method that acquires a video stream of the user's real environment using a camera on a user-worn head-mounted display device. Then, an image rendering model is trained based on the current video stream and the camera pose data corresponding to each image frame in the current video stream. This allows the trained image rendering model to render images adapted to the camera pose data. Furthermore, since camera pose data describes the camera's position when capturing an image, similarly, the user's viewpoint pose data describes the position of the user's eyes when viewing the image. The trained image rendering model then performs image rendering based on the user's viewpoint pose data at the current moment. This allows for the generation of a target rendered image adapted to the user's viewpoint, which is then displayed on a head-mounted display device. This eliminates the need for the user to remove the device; instead, the user can perceive the real-world environment through the target rendered image displayed on the head-mounted display. Furthermore, because the target rendered image is adapted to the user's viewpoint, the spatial relationships of objects in the real-world environment perceived by the user through the head-mounted display are accurate. This enables the user to accurately interact with or manipulate objects in the real-world environment based on the target rendered image displayed on the head-mounted display, such as locating and grasping objects, thus improving the user experience. Additionally, since the objects needed by the user in the real-world environment are often accidental, the target rendered image displayed on the head-mounted display in this embodiment is generated by training an image rendering model based on video streams captured by a camera of the user's real-world environment and camera pose data. The image rendering model then renders the image based on the current viewpoint pose data. Therefore, the image display method provided in this embodiment is suitable for users' accidental interaction needs; that is, users do not need to anticipate which objects in the real-world environment they will interact with before entering the virtual environment, nor do they need to know the location of the objects to be interacted with in advance.

[0108] The image display method provided in this application can be applied to various scenarios requiring virtual-real switching, and this application does not limit this. Taking a multi-party collaborative work scenario in a virtual environment as an example, User A can wear a head-mounted display device to enter the virtual environment and work collaboratively with other users. When User B asks a question, User A needs to find a real object, such as an industry data manual, to answer the question. User A can input a virtual-real switching command through voice interaction to instruct the head-mounted display device to switch to the real environment mode. In response to the user's input virtual-real switching command, the head-mounted display device displays the target rendered image to the user by pointing to the above steps S202-S208. In this way, User A can perceive their real environment through the target rendered image without removing the head-mounted display device, and thus find the industry data manual at their workstation and look up the corresponding answer through the perception of the real environment. After looking up the answer, User A can instruct the head-mounted display device to switch back to the virtual environment through voice interaction, and then User A can answer User B's question in the virtual environment.

[0109] It should be understood that the above-described multi-party collaborative work scenario is merely an illustrative example and should not be construed as limiting the application scenarios to which the image display method of this application embodiment applies.

[0110] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0111] In addition, with the above Figure 2 Corresponding to the image display method shown, this application also provides an image display device. Please refer to... Figure 6 The image display device 600 provided in one embodiment of this application is a structural schematic diagram. The device 600 may include:

[0112] The acquisition unit 610 is used to acquire the current video stream, the camera pose data corresponding to each image frame in the current video stream, and the user's viewpoint pose data at the current moment in response to a virtual-real switching event where the user switches from a virtual environment to a real environment. The current video stream is obtained by capturing the real environment in which the user is located through a camera on the head-mounted display device worn by the user, and the current video stream includes multiple image frames arranged in a time sequence.

[0113] Training unit 620 is used to train the image rendering model based on the current video stream and the camera pose data corresponding to each image frame in the current video stream;

[0114] The rendering unit 630 is used to perform image rendering processing based on the viewpoint pose data at the current moment using a trained image rendering model to obtain a target rendered image adapted to the user's viewpoint.

[0115] Display unit 640 is used to display the target rendered image through the head-mounted display device, wherein the target rendered image is used to reflect the real environment observed from the user's perspective, so that the user can perform activities in the real environment based on the target rendered image.

[0116] Optionally, when training the image rendering model based on the current video stream and the camera pose data corresponding to each image frame in the current video stream, the training unit 620 performs the following steps:

[0117] Based on multiple image frames in the first time period of the current video stream and the camera pose data corresponding to each image frame, the image rendering model is trained to obtain the first image rendering model.

[0118] Based on multiple image frames within the second time period in the current video stream and the camera pose data corresponding to each image frame, the first image rendering model is trained to obtain the second image rendering model, wherein the second time period includes the first time period.

[0119] Optionally, when the training unit 620 trains the first image rendering model to obtain the second image rendering model based on multiple image frames within the second time period in the current video stream and the camera pose data corresponding to each image frame, it performs the following steps:

[0120] Based on a preset attitude tracking algorithm and multiple image frames in the second time period, the camera attitude data corresponding to each image frame in the second time period is corrected to obtain the corrected camera attitude data corresponding to each image frame in the second time period.

[0121] Based on multiple image frames within the second time period and the corrected camera pose data corresponding to each image frame, the first image rendering model is trained to obtain the second image rendering model.

[0122] Optionally, when the training unit 620 trains the image rendering model based on multiple image frames within a first time period in the current video stream and the camera pose data corresponding to each image frame to obtain a first image rendering model, it performs the following steps:

[0123] The image rendering model performs image rendering processing based on the camera pose data corresponding to each image frame in the first time period to obtain the rendered image corresponding to each image frame in the first time period.

[0124] Based on the difference information between each image frame and the corresponding rendered image within the first time period, the model parameters of the image rendering model are adjusted to obtain the first image rendering model.

[0125] Optionally, when the training unit 620 trains the image rendering model based on multiple image frames within a first time period in the current video stream and the camera pose data corresponding to each image frame to obtain a second image rendering model, it performs the following steps:

[0126] The foreground thread trains the image rendering model based on multiple image frames within the first time period and the camera pose data corresponding to each image frame, so as to obtain the first image rendering model.

[0127] When the training unit 620 trains the first image rendering model based on multiple image frames in the second time period of the current video stream and the camera pose data corresponding to each image frame, in order to obtain the first image rendering model, it performs the following steps:

[0128] The first image rendering model is trained by a background thread based on multiple image frames within the second time period and the camera pose data corresponding to each image frame, so as to obtain the second image rendering model.

[0129] Optionally, when the rendering unit 630 performs image rendering processing based on the viewpoint pose data at the current moment using the trained image rendering model to obtain a target rendered image adapted to the user's viewpoint, it performs the following steps:

[0130] If the training of the first image rendering model in the background thread does not meet the preset training stop condition, then in the foreground thread, the first image rendering model is used to perform image rendering processing based on the viewpoint pose data at the current moment to obtain a target rendered image adapted to the user's viewpoint; or...

[0131] If the training of the first image rendering model in the background thread meets the preset training stop condition, then the first image rendering model that meets the preset training stop condition is used as the second image rendering model, and the image rendering process is performed in the foreground thread based on the viewpoint pose data at the current moment using the second image rendering model to obtain a target rendered image adapted to the user's viewpoint.

[0132] Optionally, the user's viewpoint pose data at the current moment is obtained in the following way:

[0133] The viewpoint relative pose data of the user is obtained, and the viewpoint relative pose data is used to represent the pose of the user's viewpoint relative to the camera;

[0134] The user's viewpoint pose data at the current moment is determined by multiplying the viewpoint relative pose data with the camera pose data corresponding to the image frame at the current moment in the current video stream.

[0135] Optionally, the camera pose data corresponding to each image frame in the current video stream is obtained in the following way:

[0136] Based on a preset attitude tracking algorithm, multiple image frames in the current video stream are tracked and their attitudes estimated to obtain camera attitude data corresponding to each image frame in the current video stream.

[0137] Optionally, the training unit 620 is further configured to: before training the image rendering model based on the current video stream and the camera pose data corresponding to the image frames in the current video stream, perform image recognition on the image frames at the current moment to determine whether there is an interactive object in the real environment where the user is located at the current moment; when training the image rendering model based on the current video stream and the camera pose data corresponding to each image frame in the current video stream, the training unit 620 performs the following steps: if there is an interactive object in the real environment where the user is located, then the image rendering model is trained based on the current video stream and the camera pose data corresponding to each image frame in the current video stream.

[0138] Optionally, the display unit is further configured to: after the training unit performs image recognition on the image frame at the current moment to determine whether there is an interactive object in the real environment where the user is located at the current moment, if there is no interactive object in the real environment where the user is located, then display the image frame at the current moment through the head-mounted display device.

[0139] Optionally, the virtual-to-real switching event includes at least one of the following events: the user inputs a voice command to switch the virtual reality scene to a real environment; the user triggers the virtual-to-real switching button on the head-mounted display device; the user performs a preset tapping operation on the head-mounted display device; and the user's head moves according to a preset movement pattern.

[0140] Obviously, the image display device provided in this application embodiment can serve as... Figure 2 The execution body of the image display method shown, for example Figure 2In the image display method shown, step S202 can be performed by... Figure 6 The acquisition unit 610 in the image display device 600 shown executes step S204, which can be performed by... Figure 6 The training unit 620 in the image display device 600 shown executes step S206, which can be performed by... Figure 6 The rendering unit 630 in the image display device 600 shown executes step S208, which can be performed by... Figure 6 The display unit 640 in the image display device 600 shown is activated.

[0141] According to another embodiment of this application, Figure 6 The various units in the illustrated image display device can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the image display device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0142] According to another embodiment of this application, a general-purpose computing device, such as a computer, including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), can run an application capable of performing tasks such as... Figure 2 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 6 The image display apparatus shown herein, and the image display method for implementing the embodiments of this application, are described. The computer program may be recorded on, for example, a computer-readable storage medium, and transferred through the computer-readable storage medium to the image display apparatus, and run therein.

[0143] Figure 7 This is a schematic diagram of the structure of an image display device according to an embodiment of this application. Please refer to it. Figure 7At the hardware level, the image display device includes a processor, and optionally also an internal bus, network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the image display device may also include other hardware required for other business operations.

[0144] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0145] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0146] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming an image display device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0147] In response to a virtual-to-real switching event where the user switches from a virtual environment to a real environment, the system acquires the current video stream, camera pose data corresponding to each image frame in the current video stream, and the user's viewpoint pose data at the current moment. The current video stream is obtained by capturing the real environment in which the user is located through a camera on the head-mounted display device worn by the user, and the current video stream includes multiple image frames arranged in a time sequence.

[0148] The image rendering model is trained based on the current video stream and the camera pose data corresponding to each image frame in the current video stream.

[0149] The trained image rendering model performs image rendering processing based on the viewpoint pose data at the current moment to obtain a target rendered image adapted to the user's viewpoint.

[0150] The target rendered image is displayed through the head-mounted display device, wherein the target rendered image is used to reflect the real environment as observed from the user's perspective, so that the user can perform activities in the real environment based on the target rendered image.

[0151] The above is as stated in this application. Figure 2 The method executed by the image display device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0152] The image display device can also perform Figure 2 The method, and realize the image display device in Figure 2 , Figure 4 The functions of the embodiments shown are not described in detail here.

[0153] Of course, in addition to software implementation, the image display device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0154] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable image display device including multiple applications, enable the portable image display device to perform... Figure 2 The method of the illustrated embodiment is specifically used to perform the following operations:

[0155] In response to a virtual-to-real switching event where the user switches from a virtual environment to a real environment, the system acquires the current video stream, camera pose data corresponding to each image frame in the current video stream, and the user's viewpoint pose data at the current moment. The current video stream is obtained by capturing the real environment in which the user is located through a camera on the head-mounted display device worn by the user, and the current video stream includes multiple image frames arranged in a time sequence.

[0156] The image rendering model is trained based on the current video stream and the camera pose data corresponding to each image frame in the current video stream.

[0157] The trained image rendering model performs image rendering processing based on the viewpoint pose data at the current moment to obtain a target rendered image adapted to the user's viewpoint.

[0158] The target rendered image is displayed through the head-mounted display device, wherein the target rendered image is used to reflect the real environment as observed from the user's perspective, so that the user can perform activities in the real environment based on the target rendered image.

[0159] In summary, the above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0160] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0161] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0162] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0163] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. An image display method, characterized in that, include: In response to a virtual-to-real switching event where the user switches from a virtual environment to a real environment, the system acquires the current video stream, camera pose data corresponding to each image frame in the current video stream, and the user's viewpoint pose data at the current moment. The current video stream is obtained by capturing the real environment in which the user is located through a camera on the head-mounted display device worn by the user, and the current video stream includes multiple image frames arranged in a time sequence. The image rendering model is trained based on the current video stream and the camera pose data corresponding to each image frame in the current video stream. The trained image rendering model performs image rendering processing based on the viewpoint pose data at the current moment to obtain a target rendered image adapted to the user's viewpoint. The target rendered image is displayed through the head-mounted display device, wherein the target rendered image is used to reflect the real environment as observed from the user's perspective, so that the user can perform activities in the real environment based on the target rendered image.

2. The method according to claim 1, characterized in that, The step of training the image rendering model based on the current video stream and the camera pose data corresponding to each image frame in the current video stream includes: Based on multiple image frames in the first time period of the current video stream and the camera pose data corresponding to each image frame, the image rendering model is trained to obtain the first image rendering model. Based on multiple image frames within the second time period in the current video stream and the camera pose data corresponding to each image frame, the first image rendering model is trained to obtain the second image rendering model, wherein the second time period includes the first time period.

3. The method according to claim 2, characterized in that, The step of training the first image rendering model based on multiple image frames within the second time period in the current video stream and the camera pose data corresponding to each image frame to obtain the second image rendering model includes: Based on a preset attitude tracking algorithm and multiple image frames in the second time period, the camera attitude data corresponding to each image frame in the second time period is corrected to obtain the corrected camera attitude data corresponding to each image frame in the second time period. Based on multiple image frames within the second time period and the corrected camera pose data corresponding to each image frame, the first image rendering model is trained to obtain the second image rendering model.

4. The method according to claim 2, characterized in that, The step of training an image rendering model based on multiple image frames within a first time period in the current video stream and the camera pose data corresponding to each image frame to obtain a first image rendering model includes: The image rendering model performs image rendering processing based on the camera pose data corresponding to each image frame in the first time period to obtain the rendered image corresponding to each image frame in the first time period. Based on the difference information between each image frame and the corresponding rendered image within the first time period, the model parameters of the image rendering model are adjusted to obtain the first image rendering model.

5. The method according to claim 2, characterized in that, The step of training an image rendering model based on multiple image frames within a first time period in the current video stream and the camera pose data corresponding to each image frame to obtain a first image rendering model includes: The foreground thread trains the image rendering model based on multiple image frames within the first time period and the camera pose data corresponding to each image frame, so as to obtain the first image rendering model. The step of training the first image rendering model based on multiple image frames within the second time period in the current video stream and the camera pose data corresponding to each image frame to obtain the second image rendering model includes: The first image rendering model is trained by a background thread based on multiple image frames within the second time period and the camera pose data corresponding to each image frame, so as to obtain the second image rendering model.

6. The method according to claim 5, characterized in that, The step of performing image rendering processing based on the viewpoint pose data at the current moment using the trained image rendering model to obtain a target rendered image adapted to the user's viewpoint includes: If the training of the first image rendering model in the background thread does not meet the preset training stop condition, then in the foreground thread, the first image rendering model is used to perform image rendering processing based on the viewpoint pose data at the current moment to obtain a target rendered image adapted to the user's viewpoint; or... If the training of the first image rendering model in the background thread meets the preset training stop condition, then the first image rendering model that meets the preset training stop condition is used as the second image rendering model, and the image rendering process is performed in the foreground thread based on the viewpoint pose data at the current moment using the second image rendering model to obtain a target rendered image adapted to the user's viewpoint.

7. The method according to claim 1, characterized in that, The user's viewpoint pose data at the current moment is obtained in the following way: The viewpoint relative pose data of the user is obtained, and the viewpoint relative pose data is used to represent the pose of the user's viewpoint relative to the camera; The user's viewpoint pose data at the current moment is determined by multiplying the viewpoint relative pose data with the camera pose data corresponding to the image frame at the current moment in the current video stream.

8. The method according to claim 1, characterized in that, The camera pose data corresponding to each image frame in the current video stream is obtained in the following way: Based on a preset attitude tracking algorithm, multiple image frames in the current video stream are tracked and their attitudes estimated to obtain camera attitude data corresponding to each image frame in the current video stream.

9. The method according to claim 1, characterized in that, Before training the image rendering model based on the current video stream and the camera pose data corresponding to each image frame in the current video stream, the method further includes: Image recognition is performed on the image frame at the current moment to determine whether there is an object to be interacted with in the real environment where the user is currently located. The step of training the image rendering model based on the current video stream and the camera pose data corresponding to each image frame in the current video stream includes: If there is an object to be interacted with in the user's real environment, the image rendering model is trained based on the current video stream and the camera pose data corresponding to each image frame in the current video stream.

10. The method according to claim 9, characterized in that, After performing image recognition on the image frame at the current moment to determine whether there is an object to be interacted with in the user's real environment at the current moment, the method further includes: If there is no object to be interacted with in the user's real environment, the image frame of the current moment is displayed through the head-mounted display device.

11. The method according to any one of claims 1 to 10, characterized in that, The virtual-to-real switching event includes at least one of the following events: the user inputs a voice command to switch the virtual reality scene to the real environment; the user triggers the virtual-to-real switching button on the head-mounted display device; the user performs a preset tapping operation on the head-mounted display device; the user's head moves according to a preset movement pattern.

12. An image display device, characterized in that, include: The acquisition unit is used to acquire the current video stream, the camera pose data corresponding to each image frame in the current video stream, and the user's viewpoint pose data at the current moment in response to a virtual-real switching event where the user switches from a virtual environment to a real environment. The current video stream is obtained by capturing the real environment in which the user is located through a camera on the head-mounted display device worn by the user, and the current video stream includes multiple image frames arranged in a time sequence. The training unit is used to train the image rendering model based on the current video stream and the camera pose data corresponding to each image frame in the current video stream; The rendering unit is used to perform image rendering processing based on the viewpoint pose data at the current moment using a trained image rendering model to obtain a target rendered image adapted to the user's viewpoint. A display unit is configured to display the target rendered image via the head-mounted display device, wherein the target rendered image reflects the real environment as observed from the user's perspective, so that the user can perform activities in the real environment based on the target rendered image.

13. An image display device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the image display device, the image display device is enabled to perform the method as described in any one of claims 1 to 11.

Images