Image display method and device, storage medium, equipment and program product

Abstract

The invention discloses an image display method and device, a storage medium, equipment and a program product, and the method comprises the steps: displaying a three-dimensional environment generated by virtual reality equipment, and in response to the motion of a current object wearing the virtual reality equipment, presenting a dynamic visual compensation element in the three-dimensional environment, the dizziness generated when the user uses the virtual reality equipment can be effectively reduced, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly to an image display method, apparatus, storage medium, device, and program product. Background Art

[0002] Virtual reality technology has become an important part of today's technology and entertainment fields. However, when users experience virtual reality scenarios using virtual reality devices, they often get dizzy easily because their external view of the world is eliminated and completely replaced by a simulated view that is affected by body movements but not fully controlled. This sense of dizziness is mainly caused by the inconsistency between the simulated view and the expectations of the brain based on other sensory stimuli (especially the vestibular system), resulting in a poor user experience. Summary of the Invention

[0003] Embodiments of this application provide an image display method, apparatus, storage medium, device, and program product, which can effectively reduce the dizziness generated when users use virtual reality devices and improve the user experience.

[0004] On the one hand, embodiments of this application provide an image display method, which includes:

[0005] Display a three-dimensional environment generated by a virtual reality device;

[0006] In response to the movement of the current object wearing the virtual reality device, present dynamic visual compensation elements in the three-dimensional environment.

[0007] On the other hand, embodiments of this application provide an image display apparatus, which includes:

[0008] A first display unit for displaying a three-dimensional environment generated by a virtual reality device;

[0009] A second display unit for presenting dynamic visual compensation elements in the three-dimensional environment in response to the movement of the current object wearing the virtual reality device.

[0010] On the other hand, embodiments of this application provide a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded by a processor to execute the image display method described in any of the above embodiments.

[0011] On the other hand, embodiments of this application provide a terminal device, which includes a processor and a memory. The memory stores a computer program, and the processor is used to execute the image display method described in any of the above embodiments by calling the computer program stored in the memory.

[0012] On the other hand, an embodiment of the present application provides a computer program product, including a computer program which, when executed by a processor, implements the image display method described in any one of the foregoing embodiments.

[0013] In an embodiment of the present application, a three-dimensional environment generated by a virtual reality device is displayed, and in response to the movement of the current object wearing the virtual reality device, dynamic visual compensation elements are presented within the three-dimensional environment. By presenting dynamic visual compensation elements within the three-dimensional environment in response to the movement of the current object wearing the virtual reality device, the dizziness generated by the user when using the virtual reality device can be effectively reduced, improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0015] Figure 1 It is a schematic flowchart of the image display method provided by an embodiment of the present application.

[0016] Figure 2 It is a schematic diagram of the first application scenario of the image display method provided by an embodiment of the present application.

[0017] Figure 3 It is a schematic diagram of the second application scenario of the image display method provided by an embodiment of the present application.

[0018] Figure 4 It is a schematic diagram of the third application scenario of the image display method provided by an embodiment of the present application.

[0019] Figure 5 It is a schematic diagram of the third application scenario of the image display method provided by an embodiment of the present application.

[0020] Figure 6 It is a schematic structural diagram of the image display device provided by an embodiment of the present application.

[0021] Figure 7 It is a schematic diagram of the first structure of the terminal device provided by an embodiment of the present application.

[0022] Figure 8 It is a schematic diagram of the second structure of the terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0024] The embodiments of the present application provide an image display method, device, computer-readable storage medium, terminal device, server, and computer program product. Specifically, the image display method in the embodiments of the present application can be executed by a terminal device or by a server.

[0025] The embodiments of the present application can be applied to various application scenarios such as extended reality (XR), virtual reality (VR), augmented reality (AR), and mixed reality (MR).

[0026] First, some nouns or terms that appear during the description of the embodiments of the present application are explained as follows:

[0027] A virtual reality scene is a virtual reality scene displayed (or provided) when an application runs on a terminal or a server. Optionally, the virtual reality scene is a simulation environment of the real world, or a semi-simulated and semi-fictional virtual environment. The virtual reality scene is any one of a two-dimensional virtual scene and a three-dimensional virtual scene. The virtual environment can be the sky, land, ocean, etc. Among them, the land includes environmental elements such as deserts and cities. Among them, the virtual reality scene is a scene with a complete game logic for a virtual character controlled by a user.

[0028] A virtual character refers to a dynamic object that can be controlled in a virtual reality scenario. Optionally, the dynamic object can be a virtual human, a virtual animal, an anime character, etc. The virtual character is a character controlled by the user through an input device, or an artificial intelligence (AI) set in a virtual environment battle through training, or a non-player character (NPC) set in a virtual reality scenario battle. Optionally, the virtual character is a virtual human competing in a virtual reality scenario. Optionally, the number of virtual characters in the virtual reality scenario battle is preset or dynamically determined according to the number of clients joining the battle. This application embodiment does not limit this. In a possible implementation, the user can control the virtual character to move in the virtual reality scenario. For example, the user can control the virtual character to move its head, run, jump, crawl, etc., and can also control the virtual character to use skills, virtual props, etc. provided by the application program to fight with other virtual characters.

[0029] Extended Reality (XR) is a concept that includes Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), representing an environment that connects the virtual world and the real world, and a technology that enables users to interact with this environment in real time.

[0030] Virtual Reality (VR) is a technology for creating and experiencing virtual worlds. It computationally generates a virtual environment, which is a multi-source information (the virtual reality mentioned in this article includes at least visual perception, and may also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.). It realizes the integration of the virtual environment, an interactive three-dimensional dynamic visual scene, and the simulation of entity behaviors, enabling users to immerse themselves in the simulated virtual reality environment and realizing applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisting in manufacturing, maintenance, and repair.

[0031] Augmented Reality (AR) is a technology that, during the process of a camera capturing an image, calculates the camera pose parameters of the camera in the real world (or three-dimensional world, real world) in real time, and adds virtual elements to the image captured by the camera according to these camera pose parameters. Virtual elements include, but are not limited to: images, videos, and 3D models. The goal of AR technology is to interact by overlaying the virtual world on the real world on the screen.

[0032] Mixed Reality (MR) is an artificial scene that integrates sensory inputs created by a computer (e.g., virtual objects) with sensory inputs from a physical scene or their representations. In some MR scenes, the sensory inputs created by the computer can adapt to changes in the sensory inputs from the physical scene. Additionally, some electronic systems for presenting MR scenes can monitor the orientation and / or position relative to the physical scene so that virtual objects can interact with real objects (i.e., physical elements from the physical scene or their representations). For example, the system can monitor movement so that a virtual plant appears stationary relative to a physical building.

[0033] Augmented Virtuality (AV): An AV scene refers to an artificial scene in which a computer-created scene or virtual scene incorporates at least one sensory input from a physical scene. One or more sensory inputs from the physical scene can be representations of at least one feature of the physical scene. For example, a virtual object can present the color of a physical element captured by one or more imaging sensors. Another example is that a virtual object can present features consistent with the actual weather conditions in the physical scene, such as those identified via weather-related imaging sensors and / or online weather data. In another example, an augmented reality forest can have virtual trees and structures, but animals can have features precisely reproduced from images of physical animals.

[0034] Virtual field of view refers to the area in the virtual environment that a user can perceive through lenses in a virtual reality device. The field of view angle (Field Of View, FOV) of the virtual field of view is used to represent the perceived area.

[0035] A virtual reality device is a terminal that realizes virtual reality effects. It can usually be provided in the form of glasses, a head-mounted display (HMD), contact lenses, etc., for visual perception and other forms of perception. Of course, the form of the virtual reality device is not limited to this, and it can be further miniaturized or enlarged according to needs.

[0036] The virtual reality devices described in the embodiments of this application can include, but are not limited to, the following types:

[0037] A PC-based virtual reality (PCVR) device uses the PC to perform relevant calculations and data output for virtual reality functions. An external PCVR device uses the data output by the PC to achieve virtual reality effects.

[0038] A mobile virtual reality device supports setting a mobile terminal (such as a smartphone) in various ways (such as a head-mounted display with a dedicated card slot). Through a wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations for virtual reality functions and outputs data to the mobile virtual reality device. For example, watch virtual reality videos through an APP on the mobile terminal.

[0039] An all-in-one virtual reality device has a processor for performing relevant calculations for virtual functions, thus having independent virtual reality input and output functions and not requiring connection to a PC or mobile terminal, with high freedom of use.

[0040] The following will be described in detail respectively. It should be noted that the description order of the following embodiments does not limit the priority order of the embodiments.

[0041] Each embodiment of the present application provides an image display method. This method can be executed by a terminal or a server, or jointly executed by a terminal and a server. In the embodiments of the present application, the case where the image display method is executed by a terminal (terminal device) is taken as an example for description.

[0042] Please refer to Figures 1 to 5 , Figure 1 which is a schematic flowchart of the image display method provided by the embodiments of the present application, Figures 2 to 5 and are all schematic diagrams of application scenarios of the image display method provided by the embodiments of the present application. This method can be applied to a virtual reality device. This method includes the following steps 110 to 120:

[0043] Step 110, display a three-dimensional environment generated by the virtual reality device.

[0044] Among them, the three-dimensional environment generated by the virtual reality device includes not only the real space environment but also a virtual space environment. Such a three-dimensional environment can be generated by a computer or obtained by collecting and processing through other devices. In the virtual space environment, the user can see his own and other objects' performances in the virtual space and have an immersive feeling.

[0045] Step 120, in response to the movement of the current object wearing the virtual reality device, present dynamic visual compensation elements in the three-dimensional environment.

[0046] Among them, the visual compensation element is a virtual element that performs reverse visual compensation on the visual signal generated by the movement of the current object based on visual optical flow.

[0047] For example, the current object is the user currently using the virtual reality device.

[0048] For example, when the current user wears a virtual reality device to exercise, the visual compensation element will present dynamic changes in the three-dimensional environment. This visual compensation element is realized through visual optical flow technology. Visual optical flow is a set of mutually related image sequences that can describe the movement direction and speed of a moving object. By processing and analyzing these image sequences, the movement information of the current user can be obtained, and corresponding visual signals can be generated. These visual signals are inversely applied to the visual compensation element, making it present a dynamic visual effect in the three-dimensional environment. This dynamic visual compensation element can effectively reduce the dizziness felt by the user when using the virtual reality device. This is because when the user is moving, the brain will receive corresponding visual signals, thus better adapting to the movement changes in the virtual environment. This helps to improve the user experience and enables the user to use the virtual reality device more comfortably.

[0049] In the embodiment of the present application, by responding to the movement of the current user wearing a virtual reality device, a dynamic visual compensation element is presented in the three-dimensional environment, which can effectively reduce the dizziness felt by the user when using the virtual reality device and improve the user experience.

[0050] In some embodiments, step 120 can be implemented through the following steps 121 to 125 (not shown in the figure), specifically:

[0051] Step 121: Obtain the pose data generated when the current user wears the virtual reality device to exercise, where the pose data includes head movement information and virtual movement information independent of the head movement information.

[0052] In some embodiments, the virtual reality device includes a head-mounted display for the head of the current user to wear, and a controller for the current user to interact with the virtual reality scene;

[0053] The obtaining of the pose data generated when the current user wears the virtual reality device to exercise includes:

[0054] Obtain the head movement information through the head-mounted display, where the head movement information is used to control the head pose of the virtual character corresponding to the current user in the virtual reality scene;

[0055] Obtain the virtual movement information independent of the head movement information through the controller, where the virtual movement information is used to control the body pose of the virtual character corresponding to the current user in the virtual reality scene.

[0056] For example, the virtual reality device includes a head-mounted display for the head of the current user to wear, and a controller for the current user to interact with the virtual reality scene.

[0057] Among them, head motion information is obtained through a head-mounted display, and the head motion information is used to control the head pose of the virtual character corresponding to the current object in the virtual reality scene. For example, the head motion information may include head pose information and / or viewpoint motion information of the current object obtained based on the head-mounted display, head-mounted pose information of the head-mounted display, and head spatial displacement information and head spatial rotation information input based on the controller. For example, the head pose information can be obtained by tracking the position and direction of the head of the current object through a tracking system in the head-mounted display, the viewpoint motion information can be obtained by tracking the position and direction of the head of the current object and the direction of eye movement through a tracking system in the head-mounted display, and the head-mounted pose information can be obtained by tracking the position and direction of the head-mounted display through a tracking system in the head-mounted display.

[0058] Among them, virtual motion information independent of the head motion information is obtained through the controller, and the virtual motion information is used to control the body pose of the virtual character corresponding to the current object in the virtual reality scene. The virtual motion information refers to the body pose information of the virtual character corresponding to the current object, such as postures, actions, gestures, etc. These information are used to control the movement and pose of the virtual character in the virtual reality scene to make it match the pose and actions of the current object. The virtual motion information can be used for virtual reality simulation or emulation.

[0059] For example, the controller may include, but is not limited to, a joystick, a glove, a ring, a tactile feedback device, a motion capture device, etc. These devices can capture the pose and actions of the current object through built-in sensors or external devices and transmit the pose and actions of the current object to the virtual reality device.

[0060] For example, controllers such as joysticks, gloves, and rings usually have the function of pose capture and can capture the pose data of the user through built-in sensors or external devices.

[0061] For example, the joystick can capture motion information such as the displacement, angle, and rotation of the current object, the glove can capture more detailed hand actions and finger bending information of the current object, the ring can capture the ring action information of the current object, and the tactile feedback device can provide tactile feedback to enable the current object (user) to more realistically feel the objects and scenes in the virtual environment.

[0062] Step 122, generate a first scene image of the virtual reality scene simulation based on the pose data.

[0063] Head movement information, including but not limited to nodding, shaking the head, tilting the head, turning the head, etc. This head movement information is used to control the head posture of the virtual character corresponding to the current object in the virtual reality scene. For example, if the current object (user) turns the head to the right, the virtual character should also turn the head to the right accordingly to keep in line with the head movement of the current object (user).

[0064] Virtual movement information independent of head movement information, usually including body movements, gestures, walking, etc. This virtual movement information is used to control the body posture of the virtual character corresponding to the current object in the virtual reality scene. For example, if the current object (user) moves forward, the virtual character should also move forward accordingly; if the current object (user) makes a handshake gesture, the virtual character should also make the same handshake action.

[0065] Based on this pose data, a first scene image of the virtual reality scene simulation can be generated. The first scene image is based on the head movement information and the virtual movement information, and the head movement information and the virtual movement information are generated in real time based on the actions and postures of the current object (user), and can be regarded as the avatar of the current object (user) in the virtual reality scene.

[0066] Step 123, generate a visual compensation image containing dynamic visual compensation elements, where the visual compensation elements are virtual elements that perform reverse visual compensation relative to the visual signals generated by the head movement information based on visual optical flow.

[0067] Among them, in virtual reality technology, visual compensation is an important technical means aimed at reducing the dizziness and other discomfort that users may experience when using virtual reality devices. Based on the head movement information, a visual compensation image containing dynamic visual compensation elements can be generated, and the visual compensation elements are virtual elements that perform reverse visual compensation relative to the visual signals generated by the head movement information based on visual optical flow technology.

[0068] Among them, optical flow is a concept in the detection of object motion in the visual field, used to describe the motion of observed objects, surfaces, or edges caused by the motion relative to the observer. Specifically, optical flow is the instantaneous velocity of the pixel motion of a moving object in space on the observation imaging plane. The optical flow method is a method that uses the change of pixels in the time domain in the image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, so as to calculate the motion information of the object between adjacent frames. Generally speaking, optical flow is generated by the movement of foreground objects in the scene itself, the movement of the camera, or the combined movement of both. The optical flow method is very useful in pattern recognition, computer vision, and other image processing fields, and can be used for motion detection, object segmentation, calculation of collision time and object expansion, motion compensation coding, or three-dimensional measurement through object surfaces and edges, etc. According to the density of the two-dimensional vectors in the formed optical flow field, the optical flow method is divided into two types: dense optical flow and sparse optical flow. Dense optical flow is an image registration method that performs point-by-point matching on an image or a specified area, and it calculates the offset of all points on the image to form a dense optical flow field. Sparse optical flow only tracks a subset of certain points in the image. This algorithm is usually fast and reliable because it only focuses on specific points that are easy to track.

[0069] The optical flow in the embodiment of this application can be implemented based on acceleration (Acceleration, abbreviated as A), contrast (Contrast, abbreviated as C), distance from the center (Distance from center, abbreviated as D), and area (Square, abbreviated as S), and can be abbreviated as ACDS. ACDS is a technology that provides a set of optical flows to users in a virtual reality environment. Based on the optical flow, through reverse compensation when the user moves forward, backward, left, or right, the convergence of visual and vestibular acceleration perception is achieved, thereby achieving the effect of alleviating the dizziness in virtual reality.

[0070] Among them, acceleration (Acceleration, abbreviated as A) can characterize the core variable of dizziness. The reverse compensation for the acceleration A of the virtual transformation can be obtained by creating a moving dot matrix (for generating a spatial dot matrix image) or distorting the original scene picture (for generating a mask layer), etc., to obtain a visual compensation image.

[0071] Among them, contrast (Contrast, abbreviated as C) can characterize the intensity of the perception of the picture displacement. The perception of the picture displacement of the current object can be weakened by controlling the local or global contrast C between the visual compensation image and the first scene image.

[0072] Among them, the center distance (Distance from center, abbreviated as D) can represent the distance and / or included angle between the screen of the visual compensation image and the center point of the first scene image. The larger the center distance D, the more obvious the perception of speed and / or acceleration. The center distance D can be adjusted based on the acceleration A to control the mask radius reflecting the tunnel effect in the visual compensation image.

[0073] Among them, the area (Square, abbreviated as S) can represent the size of the display area of the visual compensation image. The area S is related to the above-mentioned center distance D. The more reference screens reflecting speed and / or acceleration, the more obvious the perception of speed and / or acceleration. The area S can be adjusted based on the acceleration A to control the mask area reflecting the tunnel effect in the visual compensation image.

[0074] For example, visual optical flow generates visual signals by measuring the motion speeds and directions of different parts in a scene image. In a virtual reality scene, by capturing head motion information, corresponding visual signals can be obtained. Based on these visual signals, visual compensation elements for reverse visual compensation relative to the visual signals generated by the head motion information can be generated.

[0075] The visual signals generated by the visual compensation elements are opposite to the visual signals generated by the head motion information. For example, if the head of the current object (user) moves forward, the dynamic visual compensation elements in the visual compensation image may move backward to offset the visual changes caused by the forward movement of the head. This compensation can reduce dizziness and other discomfort that may be caused by the inconsistency between the simulated view and the visual signals expected by the brain.

[0076] In some embodiments, the visual compensation image is a see-through visual compensation image.

[0077] Among them, the see-through visual compensation image can make the visual compensation image transparent or semi-transparent to some extent. In subsequent steps, when the visual compensation image is displayed in the first scene image, it enables the user to more clearly see other objects and scenes in the first scene image through the visual compensation image, enhancing the user's immersion and experience quality.

[0078] In some embodiments, the visual compensation image includes a spatial dot matrix diagram;

[0079] Generating the visual compensation image including dynamic visual compensation elements includes:

[0080] Generating a spatial dot matrix diagram including dynamic visual compensation elements based on the head motion information.

[0081] For example, the head movement information may include the head pose information and / or the viewpoint movement information of the current object obtained based on the head-mounted display, the head-mounted pose information of the head-mounted display, and the head spatial displacement information and head spatial rotation information input based on the controller. Based on the head movement information, a spatial dot matrix image including dynamic visual compensation elements is generated, and the spatial dot matrix image is an image for reverse visual compensation with respect to the visual signal generated based on the head movement information based on visual optical flow.

[0082] Among them, the head spatial displacement information and head spatial rotation information in the head movement information may have a proportional mapping relationship with image attributes such as the optical flow direction, moving speed, and dot matrix area of the visual compensation elements in the spatial dot matrix image.

[0083] For example, the visual compensation image includes a spatial dot matrix image. The spatial dot matrix image is a special image composed of many points, and the points can represent visual compensation elements. Each point can independently set attributes such as color, size, and shape. Such an image can be used to generate target visual effects, such as magnification, reduction, rotation, three-dimensional sense, sense of distance, sense of perspective, etc. The spatial dot matrix image can be composed of an outer dot matrix or an inner dot matrix. The target visual effect can be generated by controlling the brightness, transparency, saturation, area, and volume of the dot matrix.

[0084] For example, as Figure 2 shown in the schematic diagram of the spatial dot matrix image, a spatial dot matrix image 20 including dynamic visual compensation elements is generated based on the head movement information, and the spatial dot matrix image 20 may be composed of a plurality of points 21 representing visual compensation elements.

[0085] For example, a perspective spatial dot matrix image including dynamic visual compensation elements is generated. For example, the perspective effect of the spatial dot matrix image 20 can be achieved through the gaps between the points 21 (visual compensation elements) in the spatial dot matrix image 20. For example, the perspective effect of the spatial dot matrix image 20 can also be achieved by globally or locally adjusting the transparency and saturation of the spatial dot matrix image 20. For example, the perspective effect of the spatial dot matrix image 20 can also be achieved by adjusting the transparency and saturation of some or all of the points 21 (visual compensation elements) in the spatial dot matrix image 20.

[0086] For example, when generating a spatial dot pattern containing dynamic visual compensation elements, the motion attributes of the spatial dot pattern can be adjusted according to the head motion information. For example, if the head of the current object moves forward, the dots (visual compensation elements) of the spatial dot pattern may move backward to offset the visual changes caused by the forward movement of the head; if the head of the current object moves backward, the dots (visual compensation elements) of the spatial dot pattern may move forward to offset the visual changes caused by the backward movement of the head; if the head of the current object moves left, the dots (visual compensation elements) of the spatial dot pattern may move right to offset the visual changes caused by the leftward movement of the head; if the head of the current object moves right, the dots (visual compensation elements) of the spatial dot pattern may move left to offset the visual changes caused by the rightward movement of the head. This compensation can reduce dizziness and other discomfort that may be caused by the inconsistency between the simulated view and the visual signals expected by the brain.

[0087] In addition, by adjusting the number, position, size, spacing, and motion attributes of each dot (visual compensation element) in the spatial dot pattern, target visual effects such as three-dimensional sense, sense of distance, and sense of perspective can be generated. The visual signals generated by the spatial dot pattern for the brain are consistent with the visual signals generated by the vestibular system of the current object based on the head movement information.

[0088] In some embodiments, the visual compensation image includes a mask layer;

[0089] Generating a visual compensation image containing dynamic visual compensation elements based on the head motion information includes:

[0090] Generating a mask layer containing dynamic visual compensation elements based on the head motion information.

[0091] For example, in order to achieve more accurate visual compensation, the visual compensation image may include a mask layer. The mask layer can be generated according to the head motion information and can adapt to different head motions and perspective changes. When generating a mask layer containing dynamic visual compensation elements, the view of the mask layer can be adjusted and transformed based on the head motion information, such as the position, orientation, and posture of the head. This transformation can be achieved by using image transformation algorithms, such as affine transformation or perspective transformation, to achieve the perspective effect of the mask layer.

[0092] Among them, the visual compensation elements in the mask layer can also be generated based on visual optical flow. In a head-mounted display, the visual optical flow can generate visual compensation elements for reverse visual compensation relative to the visual signals generated by the captured head motion information, and present the visual compensation elements in the mask layer. This reverse visual compensation can eliminate visual blur and distortion caused by head motion and provide a clearer and more natural visual effect.

[0093] For example, a see-through mask layer containing dynamic visual compensation elements is generated. For instance, the see-through effect of the mask layer can also be achieved by globally or locally adjusting the transparency and saturation of the mask layer.

[0094] In some embodiments, generating the visual compensation image containing dynamic visual compensation elements includes:

[0095] Generating a visual compensation image containing visual compensation elements based on the head movement information and the first scene image, where the visual compensation elements have visual attribute features matching those of the first scene image.

[0096] Among them, in order to achieve a more realistic virtual effect, a visual compensation image containing visual compensation elements can be generated based on the head movement information and the first scene image. Reverse visual compensation technology can be used to eliminate the visual blurring and distortion caused by head movement, providing a clearer and more natural visual effect. This reverse visual compensation technology can be generated based on visual optical flow. By capturing the head movement information and generating a visual signal relative to the head movement information for reverse visual compensation, the visual compensation elements can be better integrated into the first scene image.

[0097] The generation process of such a visual compensation image can include the following steps:

[0098] 1) Analyze the first scene image to obtain the visual attribute features of the first scene image, such as color, texture, light and shadow, etc.

[0099] 2) Generate visual compensation elements matching the first scene image based on the head movement information and the visual attribute features of the first scene image. These visual compensation elements have visual attribute features matching those of the first scene image. For example, these visual compensation elements can be virtual objects, characters, symbols, etc., and the visual compensation elements can be seamlessly integrated with the first scene image.

[0100] 3) Synthesize the visual compensation elements with a template image to generate a visual compensation image containing dynamic visual compensation elements. For example, the visual compensation elements in such a visual compensation image can change with the movement and perspective change of the head, providing a more realistic and natural visual experience.

[0101] For example, the template image can be a spatial lattice diagram, and the visual compensation elements form the points in the spatial lattice diagram. As Figure 3 shown in the schematic diagram of the application scenario, the first scene image 10 is a desert virtual scene, and the points 21 in the generated visual compensation image 20 are visual compensation elements representing dust. As Figure 4The schematic diagram of the application scenario shown, the first scene image 10 is a virtual scene of a night forest, and the point 21 in the generated visual compensation image 20 is a visual compensation element representing a firefly.

[0102] For example, the template image can be a mask layer, and the visual compensation element forms an image block or a virtual object in the mask layer.

[0103] For example, by generating a visual compensation image containing visual compensation elements based on the head movement information and the first scene image, and the visual compensation elements have matching visual attribute features with the first scene image, the user can see the scene image containing the visual compensation elements in the head-mounted display, and these visual compensation elements can be seamlessly integrated with the first scene image, making the user feel that the visual compensation elements are really existing. This technology can be widely applied to fields such as virtual reality, augmented reality, games, training, etc., to provide a more personalized and immersive experience.

[0104] In some embodiments, the visual attribute features include at least one of spatial features, color features, brightness features, detail texture features, and motion features.

[0105] For example, the visual attribute features can include at least one of spatial features, color features, brightness features, detail texture features, and motion features. The visual compensation elements have matching visual attribute features with the first scene image so that the visual compensation elements can be integrated into the first scene image.

[0106] For example, the spatial features match: the visual compensation elements are consistent with the spatial layout and geometric features of the first scene image, so that the visual compensation elements have matching visual attribute features with the first scene image. If the spatial features of the visual compensation elements do not match those of the first scene image, there will be a visual conflict between the visual compensation elements and the first scene image, destroying the overall consistency.

[0107] For example, the color features match: the visual compensation elements match the color features of the first scene image, so that the visual compensation elements have matching visual attribute features with the first scene image. If the color of the visual compensation elements is significantly different from that of the first scene image, there will be a visual separation between the visual compensation elements and the first scene image, destroying the overall sense of fusion.

[0108] For example, the brightness features match: the visual compensation elements match the brightness features of the first scene image, so that the visual compensation elements have matching visual attribute features with the first scene image. If the brightness of the visual compensation elements is significantly different from that of the first scene image, there will be a visual separation between the visual compensation elements and the first scene image, destroying the overall sense of fusion.

[0109] For example, the detailed texture features match: the visual compensation element has detailed texture features that match the first scene image. If the visual compensation element does not match the detailed texture features of the first scene image, there will be visual disharmony between the visual compensation element and the first scene image, destroying the overall sense of integration.

[0110] For example, the motion features match: if the object or element in the first scene image is in motion, then the visual compensation element also needs to maintain the same or similar motion trajectory and speed to maintain the consistency of the motion features.

[0111] Therefore, the visual compensation element and the first scene image being consistent or matching in visual attribute features such as spatial features, color features, brightness features, detailed texture features, and motion features can achieve good fusion, thus creating a realistic feeling visually.

[0112] In some embodiments, the method further includes:

[0113] Based on the first motion direction indicated by the head motion information, determine the second motion direction of the visual compensation element, where the first motion direction is opposite to the second motion direction.

[0114] For example, the second motion direction of the visual compensation element in the visual compensation image is opposite to the first motion direction indicated by the head motion information. For example, if the head of the current object (user) moves to the left, the visual compensation element in the visual compensation image will move to the right to offset the visual changes caused by the head moving forward, which can effectively reduce dizziness and other discomfort that may be caused by the inconsistency between the simulated view and the visual signals expected by the brain.

[0115] For example, the second motion direction of the visual compensation element in the visual compensation image is opposite to the first motion direction indicated by the head motion information. For example, if the head of the current object (user) moves forward, the visual compensation element in the visual compensation image will move backward. In addition to offsetting the visual changes caused by the head moving forward, it can also create a perspective effect visually, making the user feel as if they are passing through a tunnel or entering a cave. This effect can enhance the user's immersion and provide a richer experience for the user in the virtual environment.

[0116] Step 124, display the visual compensation image in the specified area of the first scene image to generate a target scene image.

[0117] First, it is necessary to determine the display position of the visual compensation image in the first scene image, that is, to determine the specified area of the first scene image. This display position can be specified by the user or automatically calculated by an algorithm. The direction and position of the user's line of sight can be determined based on the tracking system in the head-mounted display, so as to display the visual compensation image at the corresponding position in the first scene image. Among them, for the virtual scene movement corresponding to different applications, the visual compensation image (such as a spatial lattice diagram, a mask layer, etc.) can take any form and appear in any specified area within the visible range of the target scene image. Usually, the visual compensation image can be placed in the peripheral visual area where the user is less sensitive to details, such as displaying the visual compensation image in the surrounding area of the first scene image. For example, it can be restricted to the outer area of the butterfly-shaped visual area corresponding to the visual center of the user's eyes. For example, the compensated visual image can be mixed on the entire grid of the first scene image. On the one hand, it can reduce visual interference, and on the other hand, it can still provide the necessary visual signals to the brain to achieve perceptual convergence.

[0118] Then, the visual compensation image can also be adjusted in terms of scaling, saturation, transparency, etc. according to the changes in the head pose and body pose of the virtual character corresponding to the current object.

[0119] Then, the adjusted visual compensation image is synthesized with the first scene image to generate the target scene image. This process can be implemented using image processing software or algorithms, such as using the pixel-level weighted average method or the region-based fusion algorithm. The visual compensation image and the first scene image can be effectively fused together to generate a natural and realistic target scene image.

[0120] At the same time, in order to provide a more realistic and natural visual experience, it is also necessary to continuously update the visual compensation image according to the changes in the pose data to respond to the movement and perspective changes of the user's head.

[0121] Among them, the visual compensation image can be a perspective visual compensation image. In the generated target scene image, the perspective visual compensation image has less occlusion of the first scene image and retains more immersion. This is because when determining the position, size, and / or transparency of the visual compensation image, the occlusion of important visual information in the first scene image can be minimized, enabling the user to more naturally and realistically experience the scene and atmosphere in the virtual environment.

[0122] Such as Figure 3The schematic diagram of the application scenario shown, the first scene image 10 is a desert virtual scene, and the point 21 in the visual compensation image 20 is a visual compensation element representing dust. The visual compensation image 20 is displayed in a specified area (such as the surrounding area) of the first scene image 10 to generate a target scene image 30, which is a desert virtual scene image with a flying dust effect.

[0123] As Figure 4 The schematic diagram of the application scenario shown, the first scene image 10 is a night forest virtual scene, and the point 21 in the visual compensation image 20 is a visual compensation element representing fireflies. The visual compensation image 20 is displayed in a specified area (such as the surrounding area) of the first scene image 10 to generate a target scene image 30, which is a night forest virtual scene image with flying fireflies effect.

[0124] In some embodiments, the specified area is the surrounding area of the first scene image, and the surrounding area is located at the edge of the central area of the first scene image;

[0125] The step of displaying the visual compensation image in the specified area of the first scene image to generate a target scene image includes:

[0126] Displaying the visual compensation image in the surrounding area of the first scene image to generate a target scene image.

[0127] For example, since most users focus their vision on the central area and are not sensitive to the peripheral visual area within the visible range of the user, in order to effectively reduce the dizziness generated when the user uses the virtual reality device to view the target scene image, the specified area can be the surrounding area of the first scene image, and the surrounding area is located at the edge of the central area of the first scene image. In addition, this selection can better utilize the central area of the first scene image and make the first scene image and the visual compensation image more naturally integrated.

[0128] In the process of displaying the visual compensation image in the specified area of the first scene image, various different algorithms and technologies can be adopted. For example, image processing software or algorithms can be used to adjust the size and position of the visual compensation image to match the surrounding area of the first scene image. Region-based fusion algorithms, etc. can also be used to fuse the visual compensation image with the first scene image to generate a target scene image.

[0129] Among them, when displaying the visual compensation image around the first scene image, it is necessary to avoid excessive occlusion or interference with the central area of the first scene image. This can maintain the importance and prominent position of the central area of the first scene image in the target scene image, while making the visual compensation image and the first scene image more naturally integrated, and at the same time effectively reducing the dizziness generated by the user when viewing the target scene image using the virtual reality device.

[0130] As Figure 5 shown in the schematic diagram of the application scenario, the visual compensation image 20 is displayed in the surrounding area of the first scene image 10 to generate the target scene image 30. Among them, the first area scene image corresponding to the central area of the first scene image 10 is displayed in the central area without occlusion, and the second area scene image corresponding to the surrounding area of the first scene image 10 is displayed in the surrounding area through the semi-transparent or transparent visual compensation image. The current object views the target scene image 30 through the head-mounted display 100 worn on the head.

[0131] Step 125, presenting the target scene image including dynamic visual compensation elements in the three-dimensional environment.

[0132] For example, dynamic visual compensation elements can be rendered and added to the target scene image. These visual compensation elements are virtual elements generated based on the visual optical flow technology and can dynamically change with the movement of the user. When presenting the target scene image, these visual compensation elements will be combined with the image to form a dynamic visual effect. By presenting the target scene image including dynamic visual compensation elements in the three-dimensional environment, a more realistic and immersive virtual reality experience can be provided for the user. At the same time, this technology can also effectively reduce the dizziness feeling generated by the user when using the virtual reality device and improve the comfort and experience of the user.

[0133] In some embodiments, the visual compensation image is a perspective visual compensation image;

[0134] The presenting the target scene image including dynamic visual compensation elements in the three-dimensional environment includes:

[0135] Presenting the target scene image including dynamic visual compensation elements in the three-dimensional environment, wherein the area scene image corresponding to the specified area of the first scene image in the target scene image is displayed through the visual compensation image.

[0136] For example, the visual compensation image is a see-through visual compensation image. This means that the visual compensation image can be regarded as a transparent or semi-transparent image that does not completely block other images or objects in the target scene image. When presenting the target scene image, the visual compensation image is superimposed or embedded in the target scene image, but it does not completely cover or block other images or objects in the target scene image. This designated area is the area covered by the visual compensation image, and the area scene image corresponding to the designated area of the first scene image can be displayed through the visual compensation image. By presenting a see-through visual compensation image, users can better understand and operate the objects and scenes in the virtual environment.

[0137] In some embodiments, presenting the target scene image including dynamic visual compensation elements in the three-dimensional environment includes:

[0138] When presenting the target scene image including dynamic visual compensation elements in the three-dimensional environment, dynamically adjusting the area size of the designated area according to the pose data;

[0139] wherein, when the pose data indicates that the head pose of the virtual character is in a moving state and the body pose is in a static state, reducing the designated area until the visual compensation image including the dynamic visual compensation elements disappears from the target scene image; and

[0140] when the pose data indicates that the head pose of the virtual character is in a moving state and the body pose is in a moving state, expanding the designated area until the visual compensation image including the dynamic visual compensation elements is completely displayed in the target scene image.

[0141] For example, when displaying the target scene image, the area size of the designated area can be dynamically adjusted according to the pose data. This dynamic adjustment can enhance the realism and immersion of the virtual environment while ensuring that users avoid dizziness, enabling users to more naturally and realistically experience the scenes and atmosphere in the virtual environment.

[0142] When the pose data indicates that the head pose of the virtual character is in a moving state and the body pose is in a static state, it indicates that the simulated view is gradually approaching the visual signal expected by the brain based on the vestibular system. At this time, the designated area can be reduced until the visual compensation image disappears from the target scene image. This reduction adjustment can reduce the occlusion of important information in the first scene image and avoid interference and dizziness caused by excessive visual compensation images to the user's line of sight.

[0143] When the attitude data indicates that the head attitude of the virtual character is in a moving state and the body attitude is in a moving state, it indicates that the simulated view is gradually deviating from the visual signals expected by the brain based on the vestibular system. At this time, the specified area can be enlarged until the visual compensation image is completely displayed in the target scene image, so as to restore the visual compensation image containing dynamic visual compensation elements, and perform reverse visual compensation on the visual signals generated by the head movement information through the visual compensation image, effectively reducing the user's sense of dizziness.

[0144] It should be noted that when dynamically adjusting the size of the specified area, it is necessary to ensure the smoothness and naturalness of the adjustment and avoid obvious jumps and discontinuities. This can be achieved by using appropriate algorithms and technologies, such as using interpolation algorithms to smoothly adjust the size of the specified area.

[0145] In some embodiments, presenting the target scene image containing dynamic visual compensation elements in the three-dimensional environment includes:

[0146] When presenting the target scene image containing dynamic visual compensation elements in the three-dimensional environment, dynamically adjust the transparency of the visual compensation image according to the attitude data;

[0147] Wherein, when the attitude data indicates that the head attitude of the virtual character is in a moving state and the body attitude is in a static state, increase the transparency of the visual compensation image until the visual compensation image containing dynamic visual compensation elements disappears in the target scene image; and

[0148] When the attitude data indicates that the head attitude of the virtual character is in a moving state and the body attitude is in a moving state, reduce the transparency of the visual compensation image until the visual compensation image containing dynamic visual compensation elements is completely displayed in the target scene image.

[0149] For example, when displaying the target scene image, the transparency of the visual compensation image can be dynamically adjusted according to the attitude data. This dynamic adjustment can enhance the realism and immersion of the virtual environment while ensuring that the user avoids dizziness, enabling the user to more naturally and realistically experience the scenes and atmosphere in the virtual environment.

[0150] When the attitude data indicates that the head attitude of the virtual character is in a moving state and the body attitude is in a static state, it indicates that the simulated view is gradually converging with the visual signals expected by the brain based on the vestibular system. At this time, the transparency of the visual compensation image can be increased until the visual compensation image disappears in the target scene image. This adjustment of increasing transparency can make the first scene image more prominent and clear, while reducing the interference and dizziness caused by the visual compensation image to the user's line of sight.

[0151] When the attitude data indicates that the head attitude of the virtual character is in a moving state and the body attitude is in a moving state, it indicates that the simulated view is gradually deviating from the visual signals expected by the brain based on the vestibular system. At this time, the transparency of the visual compensation image can be reduced until the visual compensation image is completely displayed in the target scene image, so as to restore the visual compensation image containing dynamic visual compensation elements, and perform reverse visual compensation on the visual signals generated by the head movement information through the visual compensation image, effectively reducing the user's sense of dizziness.

[0152] It should be noted that when dynamically adjusting the transparency of the visual compensation image, it is necessary to ensure the smoothness and naturalness of the adjustment to avoid obvious jumps and discontinuities.

[0153] In some embodiments, presenting the target scene image containing dynamic visual compensation elements in the three-dimensional environment includes:

[0154] When presenting the target scene image containing dynamic visual compensation elements in the three-dimensional environment, dynamically adjust the saturation of the visual compensation image according to the attitude data;

[0155] Among them, when the attitude data indicates that the head attitude of the virtual character is in a moving state and the body attitude is in a static state, reduce the saturation of the visual compensation image until the visual compensation image containing dynamic visual compensation elements disappears from the target scene image; and

[0156] When the attitude data indicates that the head attitude of the virtual character is in a moving state and the body attitude is in a moving state, increase the saturation of the visual compensation image until the visual compensation image containing dynamic visual compensation elements is completely displayed in the target scene image.

[0157] For example, when displaying the target scene image, the saturation of the visual compensation image can be dynamically adjusted according to the attitude data. This dynamic adjustment can enhance the realism and immersion of the virtual environment while ensuring that users avoid dizziness, enabling users to more naturally and realistically experience the scenes and atmospheres in the virtual environment.

[0158] When the attitude data indicates that the head attitude of the virtual character is in a moving state and the body attitude is in a static state, it indicates that the simulated view is gradually becoming consistent with the visual signals expected by the brain based on the vestibular system. At this time, the saturation of the visual compensation image can be reduced until the visual compensation image disappears from the target scene image. This adjustment of reducing saturation can make the first scene image clearer and more natural, while reducing the interference and dizziness caused by the visual compensation image to the user's line of sight.

[0159] When the attitude data indicates that the head attitude of the virtual character is in a moving state and the body attitude is in a moving state, it indicates that the simulated view gradually deviates from the visual signals expected by the brain based on the vestibular system. At this time, the saturation of the visual compensation image can be increased until the visual compensation image is completely displayed in the target scene image, so as to restore the visual compensation image containing dynamic visual compensation elements, and perform reverse visual compensation on the visual signals generated by the head movement information through the visual compensation image, effectively reducing the user's sense of dizziness.

[0160] It should be noted that when dynamically adjusting the saturation of the visual compensation image, it is necessary to ensure the smoothness and naturalness of the adjustment to avoid obvious jumps and discontinuities.

[0161] Any combination of the above technical solutions can form an optional embodiment of the present application, which will not be elaborated here one by one.

[0162] The embodiments of the present application display a three-dimensional environment generated by a virtual reality device, and present dynamic visual compensation elements in the three-dimensional environment in response to the movement of the current object wearing the virtual reality device. By presenting dynamic visual compensation elements in the three-dimensional environment in response to the movement of the current object wearing the virtual reality device, the embodiments of the present application can effectively reduce the dizziness generated by the user when using the virtual reality device, and the see-through visual compensation image has less occlusion of the first scene image, retaining more immersion and improving the user experience.

[0163] To facilitate better implementation of the image display method of the embodiments of the present application, the embodiments of the present application also provide an image display device. Please refer to Figure 6 , Figure 6 which is a schematic structural diagram of the image display device provided by the embodiments of the present application. Among them, the image display device 200 may include:

[0164] A first display unit 210 for displaying a three-dimensional environment generated by a virtual reality device;

[0165] A second display unit 220 for presenting dynamic visual compensation elements in the three-dimensional environment in response to the movement of the current object wearing the virtual reality device.

[0166] In some embodiments, the second display unit 220 includes:

[0167] An acquisition subunit for acquiring attitude data generated when the current object wears the virtual reality device and moves, where the attitude data includes head movement information and virtual movement information independent of the head movement information;

[0168] A first generation subunit for generating a first scene image of a virtual reality scene simulation based on the attitude data;

[0169] A second generation subunit, configured to generate a visual compensation image including dynamic visual compensation elements, where the visual compensation elements are virtual elements for performing reverse visual compensation relative to the visual signals generated by the head motion information based on visual optical flow;

[0170] A third generation subunit, configured to display the visual compensation image in a specified area of the first scene image to generate a target scene image including dynamic visual compensation elements;

[0171] A display subunit, configured to present the target scene image including dynamic visual compensation elements in the three-dimensional environment.

[0172] In some embodiments, the visual compensation image is a see-through visual compensation image;

[0173] The display subunit is configured to present the target scene image including dynamic visual compensation elements in the three-dimensional environment, and a regional scene image corresponding to the specified area of the first scene image in the target scene image is displayed through the visual compensation image.

[0174] In some embodiments, the visual compensation image includes a spatial dot matrix diagram;

[0175] The second generation subunit is configured to generate a spatial dot matrix diagram including dynamic visual compensation elements based on the head motion information.

[0176] In some embodiments, the visual compensation image includes a mask layer;

[0177] The second generation subunit is configured to generate a mask layer including dynamic visual compensation elements based on the head motion information.

[0178] In some embodiments, the second generation subunit is configured to generate a visual compensation image including dynamic visual compensation elements based on the head motion information and the first scene image, and the visual compensation elements have visual attribute features matching those of the first scene image.

[0179] In some embodiments, the visual attribute features include at least one of spatial features, color features, brightness features, detailed texture features, and motion features.

[0180] In some embodiments, the second generation subunit is further configured to determine a second motion direction of the visual compensation elements based on a first motion direction indicated by the head motion information, and the first motion direction is opposite to the second motion direction.

[0181] In some embodiments, the specified area is the surrounding area of the first scene image, and the surrounding area is located at the edge of the central area of the first scene image;

[0182] The third generation subunit is configured to display the visual compensation image in the surrounding area of the first scene image to generate a target scene image.

[0183] In some embodiments, the virtual reality device includes a head-mounted display for the head of the current object to wear, and a controller for the current object to interact with the virtual reality scene;

[0184] The acquisition subunit is configured to: acquire head movement information through the head-mounted display, where the head movement information is used to control the head posture of the virtual character corresponding to the current object in the virtual reality scene; acquire virtual movement information independent of the head movement information through the controller, where the virtual movement information is used to control the body posture of the virtual character corresponding to the current object in the virtual reality scene.

[0185] In some embodiments, when presenting the target scene image including dynamic visual compensation elements in the three-dimensional environment, the display subunit is further configured to dynamically adjust the area size of the specified area according to the posture data;

[0186] Wherein, when the posture data indicates that the head posture of the virtual character is in a moving state and the body posture is in a static state, the specified area is reduced until the visual compensation image including the dynamic visual compensation element disappears from the target scene image; and

[0187] When the posture data indicates that the head posture of the virtual character is in a moving state and the body posture is in a moving state, the specified area is enlarged until the visual compensation image including the dynamic visual compensation element is completely displayed in the target scene image.

[0188] In some embodiments, when presenting the target scene image including dynamic visual compensation elements in the three-dimensional environment, the display subunit is further configured to dynamically adjust the transparency of the visual compensation image according to the posture data;

[0189] Wherein, when the posture data indicates that the head posture of the virtual character is in a moving state and the body posture is in a static state, the transparency of the visual compensation image is increased until the visual compensation image including the dynamic visual compensation element disappears from the target scene image; and

[0190] When the pose data indicates that the head pose of the virtual character is in a moving state and the body pose is in a moving state, reduce the transparency of the visual compensation image until the visual compensation image containing dynamic visual compensation elements is completely displayed in the target scene image.

[0191] In some embodiments, the display subunit is further configured to dynamically adjust the saturation of the visual compensation image according to the pose data when presenting the target scene image containing dynamic visual compensation elements in the three-dimensional environment;

[0192] Wherein, when the pose data indicates that the head pose of the virtual character is in a moving state and the body pose is in a static state, reduce the saturation of the visual compensation image until the visual compensation image containing dynamic visual compensation elements disappears from the target scene image; and

[0193] When the pose data indicates that the head pose of the virtual character is in a moving state and the body pose is in a moving state, increase the saturation of the visual compensation image until the visual compensation image containing dynamic visual compensation elements is completely displayed in the target scene image.

[0194] Each unit in the above image display device 200 can be implemented in whole or in part by software, hardware, and their combination. Each of the above units can be embedded in the processor in the terminal device in hardware form or be independent of it, or can be stored in the memory in the terminal device in software form, so that the processor can call and execute the operations corresponding to each of the above units.

[0195] The image display device 200 can be integrated in a terminal or server with a memory and equipped with a processor and having computing capabilities, or the image display device 200 is the terminal or server.

[0196] In some embodiments, the present application further provides a terminal device, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0197] As Figure 7 shown, Figure 7 is a schematic structural diagram of the terminal device provided by the embodiment of the present application. The terminal device 300 can generally be provided in the form of glasses, a head-mounted display (HMD), or contact lenses for realizing visual perception and other forms of perception. Of course, the form in which the terminal device is implemented is not limited to this, and it can be further miniaturized or enlarged according to needs. The terminal device 300 may include but is not limited to the following components:

[0198] Detection module 301: Detects the user's operation commands using various sensors and acts on the virtual environment, such as continuously updating the image displayed on the display screen following the user's line of sight, to achieve the interaction between the user and the virtual scene. For example, it continuously updates the real content based on the detected rotation direction of the user's head.

[0199] Feedback module 302: Receives data from the sensors and provides real-time feedback to the user. Among them, the feedback module 302 can be used to display a graphical user interface, such as displaying the virtual environment on the graphical user interface. For example, the feedback module 302 can include a display screen, etc.

[0200] Sensor 303: On the one hand, it accepts the operation commands from the user and acts on the virtual environment; on the other hand, it provides the results generated after the operation to the user in various feedback forms.

[0201] Control module 304: Controls the sensors and various input / output devices, including obtaining the user's data (such as actions, voices) and outputting perceptual data, such as images, vibrations, temperatures, and sounds, which act on the user, the virtual environment, and the real world.

[0202] Modeling module 305: Constructs a three-dimensional model of the virtual environment, and can also include various feedback mechanisms such as sounds and touches in the three-dimensional model.

[0203] In the embodiment of the present application, a three-dimensional environment can be constructed through the modeling module 305; the detection module 301 and / or the sensor 303 respond to the movement of the current object wearing the virtual reality device; the control module 304 generates visual compensation elements; and the feedback module 302 presents the dynamic visual compensation elements in the three-dimensional environment.

[0204] In some embodiments, as Figure 8 shown, Figure 8 is another structural schematic diagram of the terminal device provided by the embodiment of the present application. The terminal device 300 further includes a processor 310 with one or more processing cores, a memory 320 with one or more computer-readable storage media, and a computer program stored on the memory 320 and executable on the processor. Among them, the processor 310 is electrically connected to the memory 320. Those skilled in the art can understand that the structural diagram of the terminal device shown in the figure does not constitute a limitation on the terminal device, and it can include more or fewer components than shown, or combine certain components, or arrange different components.

[0205] The processor 310 is the control center of the terminal device 300, connecting various parts of the entire terminal device 300 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 320, and calling the data stored in the memory 320, it executes various functions of the terminal device 300 and processes data, thereby monitoring the terminal device 300 as a whole.

[0206] In the embodiments of the present application, the processor 310 in the terminal device 300 will load the instructions corresponding to the processes of one or more application programs into the memory 320 according to the following steps, and the processor 310 will run the application programs stored in the memory 320 to implement various functions:

[0207] Display the three-dimensional environment generated by the virtual reality device; in response to the movement of the current object wearing the virtual reality device, present dynamic visual compensation elements in the three-dimensional environment.

[0208] For the specific implementation of each of the above operations, reference can be made to the previous embodiments, which will not be elaborated here.

[0209] In some embodiments, the processor 310 may include a detection module 301, a control module 304, and a modeling module 305.

[0210] In some embodiments, as Figure 8 shown, the terminal device 300 further includes: a radio frequency circuit 306, an audio circuit 307, and a power supply 308. Among them, the processor 310 is electrically connected to the memory 320, the feedback module 302, the sensor 303, the radio frequency circuit 306, the audio circuit 307, and the power supply 308 respectively. Those skilled in the art can understand that Figure 7 or Figure 8 the terminal device structure shown in

[0211] The radio frequency circuit 306 can be used to receive and transmit radio frequency signals to establish wireless communication with a network device or other terminal devices, and receive and transmit signals with the network device or other terminal devices.

[0212] The audio circuit 307 can be used to provide an audio interface between the user and the terminal device through a speaker and a microphone. The audio circuit 307 can transmit the electrical signal converted from the received audio data to the speaker, which converts it into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 307 and then converted into audio data. After the audio data is output to the processor 310 for processing, it is sent to another terminal device, for example, through the radio frequency circuit 306, or the audio data is output to the memory for further processing. The audio circuit 307 may also include an earphone jack to provide communication between the peripheral earphone and the terminal device.

[0213] The power supply 308 is used to supply power to each component of the terminal device 300.

[0214] Although Figure 7 or Figure 8 not shown in the figure, the terminal device 300 may also include a camera, a Wi-Fi module, a Bluetooth module, an input module, etc., which will not be elaborated here.

[0215] In some embodiments, the present application also provides a computer-readable storage medium for storing a computer program. The computer-readable storage medium can be applied to a terminal device or a server, and the computer program enables the terminal device or the server to execute the corresponding process in the image display method in the embodiments of the present application. For the sake of brevity, it will not be elaborated here.

[0216] In some embodiments, the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the terminal device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, enabling the terminal device to execute the corresponding process in the image display method in the embodiments of the present application. For the sake of brevity, it will not be elaborated here.

[0217] The present application also provides a computer program, which includes a computer program stored in a computer-readable storage medium. The processor of the terminal device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, enabling the terminal device to execute the corresponding process in the image display method in the embodiments of the present application. For the sake of brevity, it will not be elaborated here.

[0218] It should be understood that the processor in the embodiments of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method embodiments can be completed by the integrated logic circuit in the hardware of the processor or instructions in software form. The above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0219] It can be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0220] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0221] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be described herein again.

[0222] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the function of the module or unit.

[0223] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0224] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0225] In addition, each functional unit in the embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0226] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a terminal device (which can be a personal computer or a server) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs and other various media that can store program codes.

[0227] As described above, it is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed by this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. An image display method, characterized in that, the method includes: displaying a three-dimensional environment generated by a virtual reality device; in response to the movement of the current object wearing the virtual reality device, presenting dynamic visual compensation elements in the three-dimensional environment.

2. The image display method according to claim 1, characterized in that, the presenting dynamic visual compensation elements in the three-dimensional environment in response to the movement of the current object wearing the virtual reality device includes: acquiring pose data generated when the current object wears the virtual reality device and moves, where the pose data includes head movement information and virtual movement information independent of the head movement information; generating a first scene image simulated by a virtual reality scene based on the pose data; generating a visual compensation image including dynamic visual compensation elements, where the visual compensation elements are virtual elements that perform reverse visual compensation based on the visual signal generated by the head movement information based on visual optical flow; displaying the visual compensation image in a specified area of the first scene image to generate a target scene image including dynamic visual compensation elements; presenting the target scene image including dynamic visual compensation elements in the three-dimensional environment.

3. The image display method according to claim 2, characterized in that, the visual compensation image is a see-through visual compensation image; the presenting the target scene image including dynamic visual compensation elements in the three-dimensional environment includes: presenting the target scene image including dynamic visual compensation elements in the three-dimensional environment, where the area scene image corresponding to the specified area of the first scene image in the target scene image is displayed through the visual compensation image.

4. The image display method according to claim 2, characterized in that, the visual compensation image includes a spatial dot matrix image; the generating a visual compensation image including dynamic visual compensation elements includes: generating a spatial dot matrix image including dynamic visual compensation elements based on the head movement information.

5. The image display method according to claim 2, characterized in that, the visual compensation image includes a mask layer; the generating a visual compensation image including dynamic visual compensation elements includes: generating a mask layer including dynamic visual compensation elements based on the head movement information.

6. The image display method according to any one of claims 2-5, characterized in that, the generating a visual compensation image including dynamic visual compensation elements includes: generating a visual compensation image including dynamic visual compensation elements based on the head movement information and the first scene image, where the visual compensation elements have visual attribute features matching those of the first scene image.

7. The image display method according to claim 6, characterized in that, the visual attribute features include at least one of spatial features, color features, brightness features, detail texture features, and motion features.

8. The image display method according to claim 2, characterized in that, the method further includes: Determine a second movement direction of the visual compensation element based on the first movement direction indicated by the head movement information, where the first movement direction is opposite to the second movement direction.

9. The image display method according to claim 2, wherein, the specified area is the surrounding area of the first scene image, and the surrounding area is located at the edge of the central area of the first scene image; the displaying the visual compensation image in the specified area of the first scene image to generate a target scene image includes: displaying the visual compensation image in the surrounding area of the first scene image to generate a target scene image.

10. The image display method according to claim 2, wherein, the virtual reality device includes a head-mounted display for the head of the current object to wear, and a controller for the current object to interact with the virtual reality scene; the obtaining the pose data generated when the current object wears the virtual reality device and moves includes: obtaining head movement information through the head-mounted display, where the head movement information is used to control the head pose of the virtual character corresponding to the current object in the virtual reality scene; obtaining virtual movement information independent of the head movement information through the controller, where the virtual movement information is used to control the body pose of the virtual character corresponding to the current object in the virtual reality scene.

11. The image display method according to claim 10, wherein, the presenting the target scene image including dynamic visual compensation elements in the three-dimensional environment includes: when presenting the target scene image including dynamic visual compensation elements in the three-dimensional environment, dynamically adjusting the area size of the specified area according to the pose data; wherein, when the pose data indicates that the head pose of the virtual character is in a moving state and the body pose is in a static state, reducing the specified area until the visual compensation image including the dynamic visual compensation elements disappears from the target scene image; and when the pose data indicates that the head pose of the virtual character is in a moving state and the body pose is in a moving state, enlarging the specified area until the visual compensation image including the dynamic visual compensation elements is completely displayed in the target scene image.

12. The image display method according to claim 10, wherein, the presenting the target scene image including dynamic visual compensation elements in the three-dimensional environment includes: when presenting the target scene image including dynamic visual compensation elements in the three-dimensional environment, dynamically adjusting the transparency of the visual compensation image according to the pose data; wherein, when the pose data indicates that the head pose of the virtual character is in a moving state and the body pose is in a static state, increasing the transparency of the visual compensation image until the visual compensation image including the dynamic visual compensation elements disappears from the target scene image; and When the pose data indicates that the head pose of the virtual character is in a moving state and the body pose is in a moving state, reduce the transparency of the visual compensation image until the visual compensation image containing dynamic visual compensation elements is completely displayed in the target scene image.

13. The image display method according to claim 10, wherein, presenting the target scene image containing dynamic visual compensation elements in the three-dimensional environment includes: when presenting the target scene image containing dynamic visual compensation elements in the three-dimensional environment, dynamically adjust the saturation of the visual compensation image according to the pose data; wherein, when the pose data indicates that the head pose of the virtual character is in a moving state and the body pose is in a static state, reduce the saturation of the visual compensation image until the visual compensation image containing dynamic visual compensation elements disappears from the target scene image; and when the pose data indicates that the head pose of the virtual character is in a moving state and the body pose is in a moving state, increase the saturation of the visual compensation image until the visual compensation image containing dynamic visual compensation elements is completely displayed in the target scene image.

14. An image display device, wherein, the device includes: a first display unit for displaying a three-dimensional environment generated by a virtual reality device; a second display unit for presenting dynamic visual compensation elements in the three-dimensional environment in response to the movement of the current object wearing the virtual reality device.

15. A computer-readable storage medium, wherein, the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the image display method according to any one of claims 1-13.

16. A terminal device, wherein, the terminal device includes a processor and a memory, the memory stores a computer program, and the processor is used to execute the image display method according to any one of claims 1-13 by calling the computer program stored in the memory.

17. A computer program product, including a computer program, wherein, the computer program, when executed by a processor, implements the image display method according to any one of claims 1-13.

Images