Display method, augmented reality device, storage medium and program product

Abstract

The embodiment of the invention provides a display method, augmented reality equipment, a storage medium and a program product. The method comprises the following steps that a left eye picture is displayed, a target surface of a virtual model is displayed in the left eye picture, a material texture image, in the left eye picture, of the target surface is determined through a left image, and the left image comprises a target object; a right eye picture is displayed, the target surface is displayed in the right eye picture, a material texture image of the target surface in the right eye picture is determined through a right image, and the right image comprises the target object; wherein the left image and the right image are used for forming a three-dimensional image of the target object in human eyes. According to the technical scheme provided by the embodiment of the invention, the image content with stereoscopic impression can be presented by utilizing the material texture on the surface of the virtual model, and the image browsing experience of a user can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a display method, an extended reality device, a storage medium, and a program product. Background Art

[0002] Extended Reality (XR) refers to a combination of real and virtual environments that can be interacted with by humans and machines, which is created through computer technology and wearable devices. It is a general term for various forms of reality, such as augmented reality (AR), virtual reality (VR), and mixed reality (MR). Devices that support extended display can be called extended reality devices.

[0003] In order to enrich the functions of XR devices, it is necessary to install XR applications on the XR devices. Images are a common way to present application content in XR applications. For example, images are displayed as material textures of 3D virtual models. Summary of the invention

[0004] The present application provides a display method, an XR device, a storage medium, and a program product, which are used to present three-dimensional image content using the material texture of the surface of a virtual model, thereby enhancing the user's image browsing experience.

[0005] In a first aspect of the present application, a display method is provided, which is applied to the display of an extended reality device, wherein the extended reality device supports the display of a binocular image, comprising:

[0006] Display a left-eye picture, wherein the left-eye picture displays a target surface of a virtual model, wherein a material texture image of the target surface in the left-eye picture is determined by a left image, and the left image contains a target object;

[0007] Display a right-eye picture, wherein the target surface is displayed in the right-eye picture, and a material texture image of the target surface in the right-eye picture is determined by a right image, and the right image contains the target object;

[0008] The left image and the right image are used to form a three-dimensional image of the target object in human eyes.

[0009] A second aspect of the present application provides a display method, which is applied to the display of an extended reality device, comprising:

[0010] Displaying a target surface of a virtual model, wherein a material texture image of the target surface is determined by an image, wherein the image is obtained by synthesizing a background layer image of a background environment and an object layer image of the target object;

[0011] In response to a background switching operation triggered by a user, acquiring a target background layer image of a target background environment and a target object layer image of the target object;

[0012] Determining a target composite image according to the target background layer image and the target object layer image;

[0013] Generate a transition image corresponding to a transition moment according to the background layer image, the target background layer image and the background animation configuration, wherein the transition moment is multiple;

[0014] At the transition moment, updating the material texture image of the target surface according to the transition image corresponding to the transition moment;

[0015] When the animation ends, the material texture image of the target surface is updated according to the target composite image.

[0016] A third aspect of the present application provides a display method, which is applied to the display of an extended reality device, comprising:

[0017] Display the target surface of the virtual model;

[0018] In response to an image switching operation triggered by a user, determining an exiting image and an entering image;

[0019] Generate a transition image corresponding to a transition moment according to the exit image, the entry image and the animation configuration, wherein the transition moment is multiple;

[0020] At the transition moment, updating the material texture image of the target surface according to the transition image corresponding to the transition moment;

[0021] When the animation ends, the material texture image of the target surface is updated according to the approach image.

[0022] In a fourth aspect of the present application, an electronic device is provided. The electronic device comprises: a memory and a processor, wherein:

[0023] The memory is used to store programs;

[0024] The processor is coupled to the memory, and is used to execute the program stored in the memory to implement any of the methods described above.

[0025] According to a fifth aspect of the present application, a computer-readable storage medium storing a computer program is provided, wherein the computer program can implement any of the above-described methods when executed by a computer.

[0026] In a sixth aspect of the present application, a computer program product is provided, comprising a computer program, which implements any of the methods described above when executed by a processor.

[0027] In the technical solution provided by the embodiment of the present application, the material texture image of the target surface of the virtual model in the left eye picture is determined by the left image of the target object, and the material texture image of the target surface of the virtual model in the right eye picture is determined by the right image of the target object, and the left image and the right image are used to form a three-dimensional image of the target object in the human eye. It can be seen that the technical solution provided by the embodiment of the present application can use the material texture of the surface of the virtual model to present image content with a three-dimensional sense, which can enhance the user's image browsing experience.

[0028] In the technical solution provided in the embodiment of the present application, a transition animation can be provided when the background environment of the target object is switched to enhance the user viewing experience.

[0029] In the technical solution provided in the embodiment of the present application, the material texture image of the target surface of the virtual model can be switched in the form of a transition animation to enhance the user viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 A schematic diagram of a flow chart of a display method provided in an embodiment of the present application;

[0032] Figure 2 A multi-thread management logic diagram provided for an embodiment of the present application;

[0033] Figure 3 A schematic diagram of the structure of an XR image browser provided in an embodiment of the present application;

[0034] Figure 4 A monocular image A provided in an embodiment of the present application;

[0035] Figure 5 A monocular image B provided in an embodiment of the present application;

[0036] Figure 6 A first intermediate state image provided by an embodiment of the present application;

[0037] Figure 7 A second intermediate state image provided by an embodiment of the present application;

[0038] Figure 8 A binocular image C provided in an embodiment of the present application;

[0039] Fig. 9 A binocular image D provided in an embodiment of the present application;

[0040] Fig.10 An erroneous third intermediate state image provided by an embodiment of the present application;

[0041] Fig.11 The right-eye content corresponding to the third intermediate state image provided in an embodiment of the present application;

[0042] Fig.12 The left-eye content corresponding to the third intermediate state image provided in an embodiment of the present application;

[0043] Fig.13 A fourth intermediate state image provided by an embodiment of the present application;

[0044] Fig.14 A schematic diagram of a flow chart of a display method provided in another embodiment of the present application;

[0045] Fig.15 A left-eye picture or a right-eye picture provided in an embodiment of the present application;

[0046] Fig.16 A flowchart of a display method provided in yet another embodiment of the present application;

[0047] Fig.17 A structural block diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below according to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0049] In addition, some of the processes described in the specification, claims and the above-mentioned figures of the present application include multiple operations that appear in a specific order, and these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0050] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0051] First, the vocabulary involved in the embodiments of the present application is explained. It can be understood that this explanation is for a clearer understanding of the embodiments of the present application, and does not necessarily constitute a limitation on the embodiments of the present application.

[0052] 3D Mesh: refers to the mesh of a 3D model, which is made up of multiple triangles. A 3D model is a representation of an object displayed in three dimensions by a computer or video device. It can be a real-world entity or a fictional object. These models are usually composed of polygons, which are made up of multiple triangles. In three-dimensional space, the collection of points and edges that make up these triangles is a 3D Mesh.

[0053] Binocular stereo technology: refers to the technology that simulates the principle of obtaining stereoscopic perception of the human eye, provides left and right eye images with visual differences in XR devices, allowing people to obtain a three-dimensional sense of images.

[0054] Portable Network Graphics (PNG) is a bitmap format that uses a lossless compression algorithm.

[0055] High Efficiency Image Format (HEIC) is a file format used to store images.

[0056] The Joint Photographic Experts Group (JPEG) format is an image file format.

[0057] Figure 1 A flowchart of a display method for an XR device provided in an embodiment of the present application. The method is applied to the display of an extended reality device, and the extended reality device supports the display of binocular images. Exemplarily, the XR device includes: a left display screen corresponding to the left eye and a right display screen corresponding to the right eye. Exemplarily, an XR application is running on the XR device, and the XR device uses the XR application to implement the display method provided in an embodiment of the present application. Figure 1 As shown, the method comprises:

[0058] 101. Display the left eye picture.

[0059] The left-eye picture displays a target surface of a virtual model, and a material texture image of the target surface in the left-eye picture is determined by a left image, and the left image contains a target object.

[0060] 102. Display the right eye picture.

[0061] The target surface is displayed in the right-eye picture, and the material texture image of the target surface in the right-eye picture is determined by the right image, and the right image contains the target object.

[0062] The left image and the right image are used to form a three-dimensional image of the target object in human eyes.

[0063] Exemplarily, the material texture image of the target surface in the left eye picture is determined by sampling the left image, and the material texture image of the target surface in the right eye picture is determined by sampling the right image.

[0064] Exemplarily, the left-eye image is displayed on the left display screen, and the right-eye image is displayed on the right display screen.

[0065] Exemplarily, the target surface is a plane or a curved surface.

[0066] The specific form of the above virtual model can be selected according to the actual scene requirements, and the embodiments of the present application do not make any specific limitations on this.

[0067] Optionally, the virtual model includes a 3D virtual display screen. Exemplarily, the 3D virtual display screen may be a 3D virtual curved display screen, and the target surface may be a curved display surface of the 3D virtual curved display screen. Exemplarily, the 3D virtual display screen may be a 3D virtual flat display screen, and the target surface may be a flat display surface of the 3D virtual flat display screen.

[0068] In a 3D engine, any face of any virtual model may include material attributes, and the material attributes include material texture images. Optionally, the material attributes may also include one or more of transparency, reflectivity, refractive index, luminosity, and smoothness.

[0069] The above target object can be understood as the object to be displayed, and the specific form of the object is not specifically limited in the embodiment of the present application. Taking the large-screen car viewing scene of the XR application as an example, the object can be a car to be displayed.

[0070] There is a parallax between the left image and the right image, and the parallax is used to form a three-dimensional image of the target object. The parallax is reflected in the image as a difference between the pixel position and the pixel content. That is, there is a first difference between the pixel position of the target object in the left image and the pixel position of the target object in the right image, and there is a second difference between the pixel content of the target object in the left image and the pixel content of the target object in the right image. The parallax includes the first difference and the second difference.

[0071] In order to enrich the image presentation content, the left image and the right image may also include the background environment where the target object is located, and there is a third difference between the pixel position of the background environment in the left image and the pixel position in the right image, and there is a fourth difference between the pixel content of the background environment in the left image and the pixel content in the right image, and the parallax may also include the third difference and the fourth difference. The left image and the right image are also used to form a three-dimensional image of the background environment.

[0072] It should be noted that there is also a difference between the pixel position of the virtual model in the left eye picture and the pixel position of the virtual model in the right eye picture, and this difference is used to form a three-dimensional image of the virtual model. In other words, through the XR device, the user can see the three-dimensional virtual model and the three-dimensional target object. Taking the large-screen car viewing scene of the XR application as an example, through the XR device, the user can see the three-dimensional virtual display screen and the three-dimensional car played on the virtual display screen.

[0073] Exemplarily, when the XR device supports mixed reality, that is, the XR device can integrate virtual objects (that is, virtual models) into the real world, the left-eye image and the right-eye image can also display the real environment in which the virtual model is located.

[0074] In the technical solution provided by the embodiment of the present application, the material texture image of the target surface of the virtual model in the left eye picture is determined by the left image of the target object, and the material texture image of the target surface of the virtual model in the right eye picture is determined by the right image of the target object, and the left image and the right image are used to form a three-dimensional image of the target object in the human eye. It can be seen that the technical solution provided by the embodiment of the present application can use the material texture of the surface of the virtual model to present image content with a three-dimensional sense, which can enhance the user's image browsing experience.

[0075] In order to allow the user to observe the details of the target object in all directions, the embodiment of the present application provides a solution for realizing the rotation of the three-dimensional image of the target object. The above method may also include:

[0076] 103. In response to a rotation operation triggered by a user on the target object, update the material texture images of the target surface in the left-eye image and the right-eye image according to a first preset frequency.

[0077] The orientation of the three-dimensional image is different before and after each update, that is, the three-dimensional image of the target object rotates.

[0078] The magnitude of the first preset frequency can be determined according to actual needs and / or the hardware performance of the XR device, and the embodiment of the present application does not specifically limit this. Exemplarily, the first preset frequency is 30 frames per second or 60 frames per second.

[0079] In order to allow users to experience smooth rotation of the target object, the first preset frequency is greater than or equal to a preset threshold, which may be 30 frames per second.

[0080] Exemplarily, the above 103 involves N switching times, where N is an integer greater than 1 or equal to 1.

[0081] Optionally, the material texture images of the target surface in the left-eye image and the right-eye image may be updated by material texture rendering.

[0082] Exemplarily, the XR device includes: a graphics processing unit (GPU), the GPU includes a binocular rendering shader (Shader), the binocular rendering shader re-renders the left eye texture area and the right eye texture area corresponding to the material texture of the target surface by sampling, thereby realizing the update of the material texture image of the target surface in the left eye screen and the right eye screen. Among them, the left eye texture area is used to present the material texture image of the target surface in the left eye screen, and the right eye texture area is used to present the material texture image of the target surface in the right eye screen. Among them, the binocular rendering shader can distinguish the above-mentioned left eye texture area and the right eye texture area through the application programming interface (Application Programming Interface, API) provided by the operating system of the XR device for perceiving left and right purposes. Among them, the left eye texture area and the right eye texture area both belong to the GPU texture area.

[0083] In an optional implementation, a control model is displayed in the left-eye image and the right-eye image, and the rotation operation includes a click operation or a long press operation of the user on the control model.

[0084] Exemplarily, the material texture image on the surface of the control model includes a rotation icon.

[0085] Exemplarily, the control model may be a simple 3D shape, such as a sphere or a cube, or a simple 2D plane, such as a circle or a rectangle.

[0086] When the rotation operation is a click operation of the user on the control model, the material texture images of the target surface in the left eye picture and the right eye picture are switched within a preset time length and at a first preset frequency. The preset time length can be set according to actual needs, and the embodiment of the present application does not specifically limit this, for example: 3s.

[0087] Optionally, within the preset time period, the orientation of the stereoscopic image of the target object rotates M degrees along a preset direction around a designated axis of the stereoscopic image, where the size of M can be set according to actual needs, for example: 360 degrees. The designated axis can be specified in advance, and this embodiment of the application does not specifically limit this.

[0088] When the rotation operation is a long press operation of the user on the control model, during the long press, the material texture images of the target surface in the left eye picture and the right eye picture are switched according to a first preset frequency.

[0089] Optionally, during the long press, the orientation of the stereoscopic image of the target object rotates around a designated axis of the stereoscopic image along a preset direction.

[0090] In the above embodiment, the orientation of the three-dimensional image of the target object can only rotate in one preset direction (counterclockwise rotation direction or clockwise rotation direction). For the convenience of user use, two control models can be displayed in the left-eye screen and the right-eye screen, for example: control model A and control model B. When the rotation operation is a click or long press operation of the user on control model A, the orientation of the three-dimensional image of the target object rotates in the clockwise direction around the specified axis of the three-dimensional image; when the rotation operation is a click or long press operation of the user on control model B, the orientation of the three-dimensional image of the target object rotates in the counterclockwise direction around the specified axis of the three-dimensional image.

[0091] In another optional implementation, in the above 103, “in response to a rotation operation triggered by the user on the target object, updating the material texture image of the target surface in the left-eye image and the right-eye image according to a first preset frequency” can be implemented by the following steps:

[0092] 1031. In response to the rotation operation, determine a target direction to which the three-dimensional image needs to rotate at every preset time interval.

[0093] The preset time interval and the preset frequency are reciprocals of each other.

[0094] Exemplarily, the rotation operation includes a drag operation. In response to the drag operation, at every preset time interval, the target direction to which the three-dimensional image needs to rotate is determined according to the current operation point position of the drag operation.

[0095] Exemplarily, the drag operation is a gesture operation, and the operation point position of the drag operation is also the designated point position on the user's hand. The designated point can be specified in advance, for example, it can be the point where the index finger points or the pinch point of the thumb and index finger.

[0096] Optionally, in response to the dragging operation, during the dragging process, at every preset time interval, the target direction to which the three-dimensional image needs to be rotated is determined according to the position of the operation point of the current dragging operation.

[0097] Exemplarily, the rotation direction and the angle difference may be determined based on the displacement from the starting position of the drag operation to the current operation point position of the drag operation; the angle difference refers to the difference between the target orientation and the initial orientation of the stereoscopic image. The target orientation is determined based on the initial orientation, rotation direction and angle difference of the stereoscopic image. The displacement has directionality, and the rotation direction (including counterclockwise rotation or clockwise rotation) may be determined based on the direction of the displacement, and the angle difference may be determined based on the size of the displacement. The initial orientation of the stereoscopic image refers to the orientation of the stereoscopic image at the beginning of the drag operation.

[0098] Optionally, the angle difference (eg 36 degrees) corresponding to the unit displacement size (eg 1 cm) may be determined in advance according to actual needs. In this way, the rotation angle required for the three-dimensional image may be determined later based on the displacement size and the angle difference corresponding to the unit displacement size.

[0099] 1032. Determine a target image according to the target orientation.

[0100] Exemplarily, a target image is determined according to the target orientation, and the target image includes a target object.

[0101] Exemplarily, the target image may include a target left image of the target object and a target right image of the target object, wherein the target left image of the target object includes the target object, and the target right image of the target object includes the target object, and the target left image and the target right image are used to form a three-dimensional image of the target object with the orientation being the target orientation in the human eye.

[0102] Exemplarily, the target image is a stitched image of the target left image and the target right image. For example, the first area of ​​the stitched image is the target left image, and the second area of ​​the stitched image is the target right image, wherein the stitched image can be called a binocular stereo image. The first area can be the left half area, and the second area can be the right half area.

[0103] 1033. Update the material texture image of the target surface in the left-eye image and the material texture image of the target surface in the right-eye image according to the target image.

[0104] According to the target left image, the material texture image of the target surface in the left eye picture is updated, and according to the target right image, the material texture image of the target surface in the right eye picture is updated.

[0105] The target left image and the target right image are obtained based on the target image, and the orientation of the three-dimensional image of the target object formed by the target left image and the target right image is the target orientation.

[0106] Exemplarily, the binocular rendering shader writes the target left image to the left-eye texture area corresponding to the material texture of the target surface, and writes the target right image to the right-eye texture area corresponding to the material texture of the target surface, thereby realizing the update of the material texture image of the target surface in the left-eye screen and the right-eye screen.

[0107] Exemplarily, when the target image is the above-mentioned stitched image, the first area of ​​the stitched image may be sampled to obtain a target left image, and the second area of ​​the stitched image may be sampled to obtain a target right image.

[0108] In an optional embodiment, the target object includes a first component and a second component, and the target image includes a first target layer image of the first component and a second target layer image of the second component, the first target layer image of the first component includes the first component, and the second target layer image of the second component includes the second component.

[0109] For the first part of the target object, a correspondence relationship between the first layer image of the first part and the orientation of the first part can be established in advance, and for the second part of the target object, a correspondence relationship between the second layer image of the second part and the orientation of the second part can be established. Subsequently, based on the correspondence relationship, the first layer image corresponding to the target orientation is determined as the first target layer image, and the second layer image corresponding to the target orientation is determined as the second target layer image. The first layer image of the first part includes the first part, and the second layer image of the second part includes the second part.

[0110] Taking the curved screen car viewing scene of XR application as an example, the final displayed image content is the image content synthesized by the body layer image, wheel layer image and background layer image. The car to be displayed may have a body of various colors and wheels of various styles (mainly different wheel hubs). Since the image content is obtained by synthesizing multiple layer images, the scene, body color, and wheel type can be switched by replacing the background layer image, body layer image or wheel layer image separately, which can efficiently realize the combination of multiple elements.

[0111] Assuming there are 3 types of backgrounds, 4 types of car bodies, and 5 types of wheels, if element splitting is not performed, a total of 3*4*5=60 original image materials are required; if element splitting is performed, only 3+4+5=12 original image materials are required, and 60 combinations can be displayed by subsequently superimposing multiple layers of images. In this way, not only the cost of generating image materials can be reduced, but also the storage cost of image materials can be reduced. Optionally, the final displayed image may or may not include the background environment, and this application does not make any specific restrictions on this.

[0112] In XR devices, the area available for image content presentation is much larger than that of traditional devices such as mobile phones and computers. For example, a 10m wide virtual curved screen can be placed 5m away from the user to achieve the visual effect of a large screen. In order to ensure the clarity of the image content in the display area and viewing angle of such a large screen, a higher image resolution, such as 4K or 8K, is required. Such a high resolution will not only occupy a large amount of device memory, but also bring a heavier processing (such as loading and decoding) burden.

[0113] Displaying an image on the screen generally requires the following steps:

[0114] 1. Load the image data according to the Uniform Resource Locator (URL) of the image. This step may take tens of milliseconds (local files) to several seconds (online files).

[0115] 2. Decode the image. This step may take a few milliseconds (for example, PNG) to tens of milliseconds (for example, JPEG, HEIC).

[0116] 3. Render the decoded image as a material texture to the screen. This step usually takes several milliseconds.

[0117] 4. After the screen is displayed, clear the image data and release memory resources.

[0118] It can be seen that the main time consumption for displaying the image on the screen comes from loading and decoding.

[0119] In order to shorten the delay, multithreading can be introduced to utilize the concurrent processing capability of multithreading to realize fast loading and decoding of multiple images. Optionally, in the above 1033, "updating the material texture image of the target surface in the left eye picture and the material texture image of the target surface in the right eye picture according to the target image" can be realized by the following steps:

[0120] S11, generating a first processing task, a second processing task and an update task.

[0121] The first processing task is used to output the first decoded layer image of the first target layer image to the update task.

[0122] The second processing task is used to output a second decoded layer image of the second target layer image to the update task.

[0123] The update task is used to obtain a composite image of the first decoded layer image and the second decoded layer image, and based on the composite image, update the material texture image of the target surface in the left eye picture and the material texture image of the target surface in the right eye picture.

[0124] Exemplarily, a target left image and a target right image are determined based on the synthesized image.

[0125] Exemplarily, the first area of ​​the composite image is the target left image, and the second area of ​​the composite image is the target right image.

[0126] Exemplarily, the update task is used to synthesize the first decoded layer image, the second decoded layer image and the background layer image of the background environment to obtain a synthesized image. The background layer image of the background environment includes the background environment, which refers to the background environment displayed in the left image and the right image of the target object. That is to say, in the scene of rotating the three-dimensional image of the target object, the background environment remains unchanged before and after the update.

[0127] Exemplarily, the first processing task and the second processing task may include steps such as loading, decoding, outputting and cleaning that need to be executed serially. Among them, loading refers to loading image data from a local disk or a network, decoding refers to decoding the image data to obtain a bitmap image, outputting refers to outputting the bitmap image to the update task, and the cleaning step refers to cleaning up unnecessary data to release occupied memory resources.

[0128] Exemplarily, the update task may include the steps of layer image synthesis, rendering, cleaning, etc. Layer image synthesis refers to the step of synthesizing multiple layer images, and rendering refers to writing image data into the left eye texture area and the right eye texture area mentioned in the above embodiment.

[0129] S12: Distribute the first processing task, the second processing task, and the update task to multiple threads for processing.

[0130] Exemplarily, the first processing task, the second processing task and the update task may be distributed to multiple threads through a task distribution strategy. The task distribution strategy may be set according to actual needs, and the embodiment of the present application does not impose any limitation on this.

[0131] In actual implementation, in order to optimize the efficiency of thread use and avoid repeatedly starting and ending threads, thread pool technology can be used to manage multiple threads.

[0132] Optionally, the above method may further include one or more of the following steps:

[0133] 104. Generate a first task dependency condition for the first processing task and the second processing task.

[0134] The first task dependency condition is used to limit the task to be executed after the decoding step involved in the preceding processing task is completed, and the generation time of the preceding processing task is earlier than the generation time of the first processing task and the second processing task.

[0135] The first task dependency condition is sent to the thread receiving the first processing task and the thread receiving the second processing task.

[0136] In the embodiment of the present application, it is possible to avoid the situation where the delay of the first displayed image increases due to the loading and decoding of the later displayed image occupying hardware resources.

[0137] 105. Generate a second task dependency condition for the update task.

[0138] The second task dependency condition is used to limit the task to be executed after the first processing task and the second processing task are completed.

[0139] Optionally, the second task dependency condition is used to limit the task to be executed after the output step of the first processing task and the output step of the second processing task are completed.

[0140] The second task dependency condition may be sent to the thread that receives the update task.

[0141] The following will be combined Figure 2 The following is an example of multi-thread management logic:

[0142] like Figure 2 As shown, image data loading 1, image data decoding 2, image texture output 3, and image data cleaning 4 corresponding to the image in each layer can be executed serially, and layer synthesis 1, image on-screen rendering 2, and image data cleaning can be executed serially. For different layers (including layer synthesis), threads can be opened separately to allow different layers to be loaded, decoded, and output in parallel in their respective threads, which can bring a significant improvement compared to processing all images in a single thread. Here, an operation dependency can be introduced: B. Layer synthesis dependency, that is, the multi-layer synthesis operation needs to rely on each layer to complete the image texture output, which is the second task dependency condition mentioned above.

[0143] Secondly, for different images of the same layer (for example, switching from the current image to the next image), they can also be put into a new thread for parallel accelerated loading. This can introduce three operation dependencies:

[0144] A. Internal data loading waiting: The two time-consuming operations of "loading and decoding" the next image should be performed after the "loading and decoding" of each current image to avoid occupying the hardware resources required for loading and decoding the current image, resulting in increased delay in loading the current image.

[0145] C. Image output waiting 1: The "texture output" of the next image should be performed after the "layer synthesis" corresponding to each current image. This is because it is necessary to ensure the synchronization of multiple layers to avoid incorrect synthesis such as "the next image of layer 1" superimposed on "the current image of layer 2".

[0146] D. Output Wait 2: The "texture output" of the next image should be performed after the "texture output" of each current image. This is used to avoid disordered image output caused by multi-threaded parallelism. In actual implementation, this dependency condition is generally met, because "B. Layer Composition Dependency" plus "C. Image Output Wait 1" can automatically guarantee "D. Image Output Wait 2".

[0147] Then, for the next image that needs to be layered, the synthesis can also be accelerated through multithreading. An operation dependency "E. Image rendering wait" can be introduced: the "on-screen rendering" of the next synthesized image should be performed after the "on-screen rendering" of the current synthesized image. This can avoid the disorder of image output caused by parallelism.

[0148] As an option, in order to reduce latency, an embodiment of the present application provides a preloading and / or predecoding solution.

[0149] Assume that an XR application has 10 images to display. While displaying the first image, the remaining 9 images can be loaded and decoded in the background. This way, when the user switches images, it only takes a few milliseconds to quickly render the images on the screen. If preloading is not performed, users may spend seconds waiting for the images to be downloaded, decoded, and rendered.

[0150] In actual use, only preloading can be turned on without predecoding. This is because: the overhead of the preloading function is to occupy a large memory space such as the preloaded image file to store the preloaded data, and this size is usually acceptable. As for the predecoding function, the image texture data (that is, the bitmap image) generated after the image is decoded will occupy a large amount of memory space. For example, a JPEG image with a 4K resolution is generally about 3 to 4MB in size. And the decoded image texture data, that is, the RGBA texture data, will reach about 31MB. The space occupied by 8K resolution will be 4 times that of 4K resolution, occupying more. If predecoding is turned on for multiple high-definition images, this will consume a lot of memory on the device. Using the technical solution provided in the embodiment of the present application, after turning on preloading, the image switching performance can be guaranteed. Among them, RGBA is a color space representing Red, Green, Blue and Alpha. Table 1 is the performance data after enabling preloading.

[0151] Table 1:

[0152]

[0153] Note: The data in the table is measured based on real devices.

[0154] In one embodiment, the method may further include:

[0155] 106. Preload multiple first layer images of the first component and multiple second layer images of the second component into the memory of the XR device.

[0156] The first layer image includes the first component, and the second layer image includes the second component.

[0157] The plurality of first layer images correspond one-to-one to the plurality of preset directions; and the plurality of second layer images correspond one-to-one to the plurality of preset directions.

[0158] The first area of ​​the first layer image is the first left component image, and the second area is the first right component image. The first left component image and the first right component image are used to form a three-dimensional image of the first component. The orientation of the three-dimensional image is the preset orientation corresponding to the first layer image.

[0159] The first area of ​​the second layer image is the second left component image, and the second area is the second right component image. The second left component image and the second right component image are used to form a three-dimensional image of the second component. The orientation of the three-dimensional image is the preset orientation corresponding to the second layer image.

[0160] In the above 1032, "determining the target image according to the target orientation" may include: determining a first layer image corresponding to the target orientation as the first target layer image; and determining a second layer image corresponding to the target orientation as the second target layer image.

[0161] In this way, the required layer images can be obtained from the preloaded data later, which improves loading efficiency, shortens latency, and improves switching smoothness.

[0162] Optionally, the left image and the right image further include the background environment where the target object is located.

[0163] Exemplarily, the left image and the right image are obtained by synthesizing a background layer image of the background environment and an object layer image of the target object.

[0164] Among them, the background layer image contains the background environment.

[0165] Exemplarily, the object layer image includes a target object.

[0166] Exemplarily, the object layer image includes a first initial layer image of a first component and a second initial layer image of a second component, wherein the first initial layer image includes the first component and the second initial layer image includes the second component.

[0167] Exemplarily, the first area of ​​the first initial layer image is the first left component image, and the second area is the first right component image. The first left component image and the first right component image are used to form a three-dimensional image of the first component, and the orientation of the three-dimensional image is a preset orientation corresponding to the first initial layer image.

[0168] Exemplarily, the first area of ​​the second initial layer image is the second left component image, and the second area is the second right component image. The second left component image and the second right component image are used to form a three-dimensional image of the second component, and the orientation of the three-dimensional image is a preset orientation corresponding to the second initial layer image.

[0169] Exemplarily, the lighting effect of the target object in the object layer image is adapted to the lighting effect of the background environment in the background layer image.

[0170] In this way, the lighting effect of the stereoscopic image of the target object matches the lighting effect of the background environment in which it is located, without being restricted by the lighting environment in which the XR device is located.

[0171] In some scenarios, users hope to see the visual effects of the target object in different background environments. In order to meet the needs of users to observe the target object in different background environments, the above method may also include:

[0172] 107. In response to a background switching operation triggered by the user, determine a target background environment.

[0173] Exemplarily, the left-eye picture and the right-eye picture also display a control model corresponding to at least one alternative background environment.

[0174] Exemplarily, the material texture image on the surface of the control model corresponding to each candidate background environment is a thumbnail of the background environment or a character pattern used to indicate the background name of the background environment.

[0175] Exemplarily, the control model may be a simple 3D shape, such as a sphere or a cube, or a simple 2D plane, such as a circle or a rectangle.

[0176] Exemplarily, the background switching operation includes a triggering operation for a certain control model, such as a click operation, a double-click operation, etc., and the candidate background environment corresponding to the candidate control model is determined as the target background environment.

[0177] 108. Acquire a target background layer image of the target background environment and a target object layer image of the target object.

[0178] The target background layer image includes the target background environment.

[0179] Exemplarily, the target object layer image includes the target object.

[0180] Exemplarily, the target object layer image includes a first target layer image of a first component and a second target layer image of a second component, wherein the first target layer image includes the first component and the second target layer image includes the second component.

[0181] Exemplarily, the lighting effect of the target object in the target object layer image is adapted to the lighting effect of the target background environment in the target background layer image.

[0182] Exemplarily, the first target layer image of the first component may include a first target left layer image and a first target right layer image of the first component.

[0183] Exemplarily, the first target layer image of the first component is a stitched image of the first target left layer image and the first target right layer image of the first component, the first area of ​​the stitched image is the first target left layer image, and the second area of ​​the stitched image is the first target right layer image.

[0184] Exemplarily, the second target layer image of the second component may include the second target left layer image and the first target right layer image of the second component.

[0185] Exemplarily, the second target layer image of the second component is a stitched image of the second target left layer image and the second target right layer image of the second component, the first area of ​​the stitched image is the second target left layer image, and the second area of ​​the stitched image is the second target right layer image.

[0186] Exemplarily, the target background layer image of the target background environment may include a target left background layer image and a target right background layer image of the target background environment.

[0187] Exemplarily, the target background layer image of the target background environment is a stitched image of a target left background layer image and a target right background layer image of the target background environment, the first area of ​​the stitched image is the target left background layer image, and the second area of ​​the stitched image is the target right background layer image.

[0188] 109. Determine a target composite image according to the target background layer image and the target object layer image.

[0189] Optionally, the target background layer image and the target object layer image are synthesized to obtain a target synthesized image. Synthesis in the embodiment of the present application refers to pixel synthesis.

[0190] Exemplarily, the first target layer image of the first component, the second target layer image of the second component, and the target background layer image are synthesized to obtain a target synthesized image.

[0191] Exemplarily, the target composite image includes a left approach image of the target object and a right approach image of the target object, wherein the left approach image includes the target object, and the right approach image includes the target object.

[0192] Exemplarily, the target composite image is a stitched image of a left approaching image of the target object and a right approaching image of the target object, the first area of ​​the stitched image is the left approaching image of the target object, and the second area of ​​the stitched image is the right approaching image of the target object.

[0193] 110. At a first moment, based on the target composite image, update the material texture image of the target surface in the left-eye image and the material texture image of the target in the right-eye image.

[0194] Exemplarily, according to the target composite image, a left approach image and a right approach image are determined. According to the left approach image, a material texture image of the target surface in the left eye image is updated. According to the right approach image, a material texture image of the target surface in the right eye image is updated.

[0195] Exemplarily, the binocular rendering shader writes the incoming left image to the left-eye texture area corresponding to the material texture of the target surface, and writes the incoming right image to the right-eye texture area corresponding to the material texture of the target surface, thereby realizing the update of the material texture image of the target surface in the left-eye screen and the right-eye screen.

[0196] Exemplarily, when the target composite image is the above-mentioned stitched image, the first area of ​​the stitched image may be sampled to obtain the approaching left image, and the second area of ​​the stitched image may be sampled to obtain the approaching right image.

[0197] In order to improve the user experience, the embodiment of the present application also provides a solution for implementing transition animation. Optionally, the method may also include the following steps:

[0198] 111. Determine a left background layer image and a right background layer image of the background environment according to the background layer image.

[0199] Exemplarily, when the background layer image is a spliced ​​image of a left background layer image and a right background layer image of the background environment, the background layer image is split to obtain a left background layer image and a right background layer image of the background environment.

[0200] 112. Determine a target left background layer image and a target right background layer image of the target background environment according to the target background layer image.

[0201] Exemplarily, when the target background layer image is a spliced ​​image of a target left background layer image and a target right background layer image of the target background environment, the target background image layer is split to obtain a target left background layer image and a target right background layer image of the target background environment.

[0202] 113. Generate a transition left background layer image corresponding to a transition moment according to the left background layer image, the target left background layer image and the background animation configuration.

[0203] Among them, the left background layer image is used as the exit image, and the target left background layer image is used as the entry image.

[0204] Among them, the transition moment is earlier than the first moment.

[0205] Exemplarily, the background animation configuration may include one or more of the following configuration items: center movement, size scaling, transparency change, and angle rotation.

[0206] Exemplarily, the background animation configuration may also include: animation properties, which may include: animation duration and whether it is interruptible. Wherein, whether it is interruptible refers to whether the animation can be ended during the animation process and respond to other user operations. Interruptible means that the animation can be ended during the animation process and respond to other user operations, and uninterruptible means that the animation cannot be ended during the animation process.

[0207] The transition moment may be one or more, and the present application embodiment does not specifically limit this. In order to present a smooth image switching animation effect, the more transition moments, the better.

[0208] The transition left images corresponding to different transition moments are different, and the transition right images corresponding to different transition moments are different.

[0209] The following will take center movement as an example:

[0210] For the sake of convenience, a normalized form can be used to describe the position of the center point (x, y) in the image, where (0,0) represents the center, the horizontal coordinate x takes the value [-1,1], -1 represents the leftmost side of the image, and 1 represents the rightmost side of the image. Similarly, the vertical coordinate y takes the value [-1,1], corresponding to the bottom of the image to the top of the image.

[0211] The left background layer image is the exit image A, and the target left background layer image is the entry image B. The background animation configuration includes: the center movement function of A is: x=-progress, y=0, where progress is the animation process (percentage), the value range is [0,1], and the value range of x is [-1,0]. The center movement function of B is: x=1-progress, y=0, where progress is the animation process (percentage), the value range is [0,1], and the value range of x is [0,1].

[0212] For example: when the animation process is 0 (that is, when the animation starts), the center point of A is (0,0) and the center point of B is (1,0).

[0213] When the animation process is 0.5 (that is, 50%), the center point position of A is (-0.5, 0), and the center point position of B is (0.5, 0). According to the center point position of A (-0.5, 0) and the center point position of B (0.5, 0), the left half of the generated transition left background layer image is the right half of A, and the right half of the generated transition left background layer image is the left half of B.

[0214] When the animation progress is 1 (that is, 100%, when the animation is completed), the center point of A is (-1,0) and the center point of B is (0,0).

[0215] Assuming that the animation lasts for 3 seconds and the second preset frequency is 100 times per second, after responding to the background switching operation, the animation progress of the current moment (that is, the transition moment) can be calculated every 0.01 seconds, and then the center point positions of A and B are calculated based on the current animation progress. According to the center point positions of A and B, the transition left background layer image at the current moment is determined.

[0216] For example: at 0.01 seconds after responding to the background switching operation, the animation progress is 0.01 / 3; at 0.02 seconds after responding to the background switching operation, the animation progress is 0.02 / 3; at 3 seconds after responding to the background switching operation, the animation progress is 3 / 3.

[0217] Exemplarily, there are multiple transition moments, and a preset animation duration can be obtained; multiple transition moments are determined according to the preset animation duration and the second preset frequency. For example: the number of transition moments is determined by multiplying the animation duration by the second preset frequency, and the time interval between two adjacent transition moments is the reciprocal of the second preset frequency.

[0218] 114. Generate a transition right background layer image corresponding to the transition moment according to the right background layer image, the target right background layer image and the background animation configuration.

[0219] Among them, the right background layer image is used as the exit image, and the target right background layer image is used as the entry image.

[0220] The generation process of the transition right background layer image can refer to the generation process of the transition left background layer image, and the embodiment of the present application will not go into details therein.

[0221] 115. Determine a transition left image and a transition right image corresponding to the transition moment according to the transition left background layer image and the transition right background layer image corresponding to the transition moment respectively.

[0222] In one embodiment, the transition left background layer image corresponding to the transition moment can be used as the transition left image corresponding to the transition moment, and the transition right background layer image corresponding to the transition moment can be used as the transition right image corresponding to the transition moment. In other words, the transition left and right images do not contain the target object.

[0223] 116. When the transition moment arrives, the material texture image of the target surface in the left eye image is updated according to the transition left image corresponding to the transition moment, and the material texture image of the target surface in the right eye image is updated according to the transition right image corresponding to the transition moment.

[0224] Exemplarily, when each transition moment arrives, the material texture image of the target surface in the left eye image is updated according to the transition left image corresponding to the transition moment, and the material texture image of the target surface in the right eye image is updated according to the transition right image corresponding to the transition moment.

[0225] Exemplarily, the binocular rendering shader writes the transition left image to the left-eye texture area corresponding to the material texture of the target surface, and writes the transition right image to the right-eye texture area corresponding to the material texture of the target surface, thereby realizing the update of the material texture image of the target surface in the left-eye screen and the right-eye screen.

[0226] In another embodiment, the above method may further include one or more of the following steps:

[0227] 116. Determine a left object layer image and a right object layer image of the target object according to the object layer image.

[0228] Exemplarily, the object layer image includes: a left object layer image and a right object layer image of the target object.

[0229] Exemplarily, the object layer image is a stitched image of a left object layer image and a right object layer image of the target object, a first area of ​​the stitched image is the left object layer image of the target object, and a second area is a stitched image of a right object layer image of the target object.

[0230] 117. Determine a target left object layer image and a target right object layer image of the target object according to the target object layer image.

[0231] Exemplarily, the target object layer image includes: a target left object layer image and a target right object layer image of the target object.

[0232] Exemplarily, the target object layer image is a stitched image of a target left object layer image and a target right object layer image of the target object, a first area of ​​the stitched image is the target left object layer image of the target object, and a second area of ​​the stitched image is the target right object layer image of the target object.

[0233] In the above 115, "determine the transition left image and transition right image corresponding to the transition moment according to the transition left background layer image and transition right background layer image corresponding to the transition moment respectively", may include: using the composite image of the transition left background layer image corresponding to the transition moment and the target left object layer image (or the left object layer image of the target object) of the target object as the transition left image corresponding to the transition moment, and using the composite image of the transition right background layer image corresponding to the transition moment and the target right object layer image (or the left object layer image of the target object) of the target object as the transition right image corresponding to the transition moment. That is to say, the target object is included in the transition left and right images, and the target object does not have any animation switching effect, and only the background environment involves the animation switching effect.

[0234] In another embodiment, the above method may further include the following steps:

[0235] 118. Generate a transition left object layer image corresponding to the transition moment according to the left object layer image, the target left object layer image and the object animation configuration.

[0236] The left object layer image is used as the exit image, and the target left object layer image is used as the entry image.

[0237] The object animation configuration may be the same as or different from the background animation configuration.

[0238] Optionally, the object animation configuration includes transparency changes, for example: in the first half of the entire animation process, the left object layer image gradually changes from transparency 0 to 1, and in the second half of the entire animation process, the target left object layer image gradually changes from transparency 1 to 0.

[0239] 119. Generate a transition right object layer image corresponding to the transition moment according to the right object layer image, the target right object layer image and the object animation configuration.

[0240] The right object layer image is used as an exit image, and the target right object layer image is used as an entry image.

[0241] The generation process of the transition right object layer image can refer to the generation process of the transition left object layer image, which will not be repeated here.

[0242] In the above 115, "determining the transition left image and transition right image corresponding to the transition moment according to the transition left background layer image and the transition right background layer image corresponding to the transition moment respectively" may include: synthesizing the transition left background layer image and the transition left object layer image corresponding to the transition moment to obtain the transition left image corresponding to the transition moment; synthesizing the transition right background layer image and the transition right object layer image corresponding to the transition moment to obtain the transition right image corresponding to the transition moment.

[0243] In this embodiment, the transition left and right images include a target object, and the target object involves an animation switching effect, and the background environment also involves an animation switching effect, and the animation switching effects involved in the two may be the same or different.

[0244] Optionally, the configuration items in the object animation configuration are different from the configuration items in the background animation configuration, and the animation properties in the object animation configuration are the same.

[0245] In the embodiment of the present application, a transition animation is provided when the images are switched, making the switching process more graceful and natural.

[0246] Optionally, in order to help developers conveniently present image content in XR applications, the embodiment of the present application provides an XR image browser, which is not a separate software or application, but a software development kit (SDK). The XR image browser provides basic capabilities for presenting images in XR applications. XR applications only need to design an additional graphical user interface to complete the presentation of image content in the application based on the capabilities provided by the XR image browser.

[0247] Alternatively, if Figure 3 As shown, the XR image browser may include three core modules: a loading module 301, an animation module 302, and a rendering module 303. Among them, the loading module is responsible for reading the image data and outputting a decoded image texture, such as: GPU texture. The animation module is responsible for implementing the switching transition animation. Specifically, the loading module obtains the current entry layer image and exit layer image, synthesizes them according to the animation rules, and continuously outputs the image texture of the animation process over time, such as: GPU texture. The rendering module is responsible for rendering the image on the screen and providing support for 3D shapes. Among them, the rendering of monocular and binocular images will be processed by different GPU shaders, and the image data will be written to the GPU rendering buffer (that is, the left eye texture area and the right eye texture area mentioned above). Among them, the rendering of monocular images is processed by a monocular shader, and the rendering of binocular stereo images is processed by a binocular shader.

[0248] like Figure 3As shown, the loading module is used to provide functions such as multi-layer synthesis, multi-threaded concurrent management, decoding and pre-decoding, loading and pre-loading. The animation module is used to provide animation synthesis, such as switching animation synthesis. Switching animation synthesis involves exit animation synthesis and entry animation synthesis.

[0249] In order to achieve various animation effects based on the animation module, the animation module can define the elements that need to be included in an image animation description. Among them, animation properties may include but are not limited to: "duration" and "whether it can be interrupted". Optionally, according to the specific effects required by the animation, animation components such as "center movement", "scaling", "transparency change", and "angle rotation" (that is, the configuration items mentioned above) can be set. During the animation process, the animation module will read these configuration information, and calculate the animation process based on the transition moment and the total duration of the animation, and then calculate the current value of each animation component based on the animation process, and synthesize the animation status of the process image and the exit image according to the current value of each animation component.

[0250] Usually, the image animation effect usually involves an entry image and an exit image. According to the above description, the animation states of the entry image and the exit image at the transition moment are calculated respectively, and then the image is synthesized based on the animation states of the entry image and the exit image at the transition moment to obtain the animation intermediate image (i.e., the transition image) at the transition moment, and then the image is handed over to the rendering module for rendering on the screen. Different switching logics are used for monocular images (i.e., the material texture images of the target surface in the left eye picture and the right eye picture are the same) and binocular stereo images.

[0251] Taking image translation switching as an example, suppose we need to switch between the following two monocular images, Figure 4 The image A shown exits from the left, letting Figure 5 Image B is shown approaching from the right.

[0252] Then, in the middle of the animation, the user can see the following Figure 6 and Figure 7 The intermediate state image (ie, transition image) shown is that after the exiting image A is completely translated to the left, the entering image B is connected to its right side to fill the vacancy caused by its exit.

[0253] But for binocular stereo images, the same process will cause problems. Suppose you need to switch between the following two binocular stereo images, let Figure 8 The image C shown (the left half is C1, the right half is C2) exits from the left, letting Fig. 9 The image D (the left half is D1 and the right half is D2) shown enters from the right side. The left half of the binocular stereo image corresponds to the left eye picture, and the right half of the binocular stereo image corresponds to the right eye picture.

[0254] If we refer to the animation process of the monocular image for image synthesis, we will get the following Fig.10 The intermediate state image shown, at this time, the left eye sees Fig.11 The right eye sees Fig.12 This is obviously not true.

[0255] Therefore, when animating a binocular image, it is necessary to first split it into a left image and a right image, apply the animation synthesis method to the left and right images respectively, and then merge them back into a binocular stereo image to obtain Fig.13 The intermediate state image shown. This allows both the left eye and the right eye to see the correct animation intermediate image.

[0256] In the actual implementation process, a general animation synthesis pipeline can be designed for monocular animation effects, and binocular animation is based on it, and the binocular image splitting and merging are connected before and after to obtain the binocular animation synthesis pipeline.

[0257] The above rendering module is mainly responsible for two things: one is to perform special processing on binocular stereo images to complete the rendering of the left and right eyes respectively; the other is to encapsulate this rendering capability into materials (Material), components (Component) and views (View) for external use.

[0258] For binocular rendering, a binocular rendering shader is set in the rendering module to tell the GPU how this binocular stereo image should be provided to the display for display. In this process, the binocular rendering shader needs to use the left and right eye perception API provided by the system platform to determine whether the left eye texture area or the right eye texture area needs to be rendered. For the left eye texture area, the left half area of ​​the binocular stereo image is sampled; for the right eye texture area, the right half area of ​​the binocular stereo image is sampled. In this way, the display can obtain the correct left and right eye pictures and present them to the user. In the embodiment of the present application, the material texture image of the target surface is switched by rendering the above-mentioned left eye texture area and right eye texture area respectively through the texture rendering interface provided by the GPU, rather than switching through the texture replacement interface provided by the GPU. It should be noted that the texture rendering interface provided by the GPU can provide a calling frequency that is higher than the calling frequency provided by the texture replacement interface provided by the GPU, which helps to achieve fast image switching capabilities.

[0259] With the rendering capability, the monocular and binocular rendering processes can be encapsulated in the lower layer, which is transparent to the outside world. Then, combined with the capabilities and functions of the 3D engine, the 3D engine materials (Material), components (Component) and views (View) that can be directly used are provided to the outside world. When used externally, you only need to create a 3D material / 3D component / 3D view, and then pass in the corresponding image file to present the image content in the 3D scene. Among them, the 3D material needs to be used with the 3D Mesh. After the material is attached to the 3DMesh, a 3D component can be obtained, which represents a 3D model with a 3D Mesh structure and display materials. At the same time, for ease of use, a programming interface for loading and generating 3D components from custom 3D Mesh resources can be provided, allowing upper-level developers to quickly present monocular / binocular image content on a custom 3D shape surface. For developers, a simple and easy-to-use 3D view can also be provided, which directly provides a 3D vision to the outside world by packaging the relevant operations of the 3D engine. Developers will be able to combine 3D vision like other 2D views (such as horizontal arrangement views, vertical stacking views, etc.), thereby integrating monocular images / binocular stereo images with 3D shapes into 2D content.

[0260] Fig.14 A flowchart of a display method for an XR device provided in an embodiment of the present application. The method is applied to the display of an extended reality device. Exemplarily, the XR device includes: a left display screen corresponding to the left eye and a right display screen corresponding to the right eye. Exemplarily, an XR application is running on the XR device, and the XR device uses the XR application to implement the display method provided in an embodiment of the present application. Exemplarily, Fig.14 All images involved in the corresponding embodiments are monocular images.

[0261] like Fig.14 As shown, the method may include the following steps:

[0262] 1401. Display the target surface of the virtual model.

[0263] The material texture image of the target surface is determined by an image of the target object, and the image is obtained by synthesizing a background layer image of a background environment and an object layer image of the target object.

[0264] The left display screen displays the left eye picture, and the right display screen displays the right eye picture, the material texture image of the target surface in the left eye picture is the image of the target object, and the material texture image of the target surface in the right eye picture is the image of the target object. In other words, the material texture image of the target surface in the left eye picture is the same as the material texture image of the target surface in the right eye picture.

[0265] The target object image includes the target object, the background layer image of the background environment includes the background environment, and the target object object layer image includes the target object. The image is obtained by synthesizing the background layer image of the background environment and the object layer image of the target object. That is, the target object image includes the target object and the background environment. Fig.15 As shown, the virtual model is a 3D virtual curved display screen 150 , the target surface is a curved display surface, and the image displayed on the curved display surface includes a microwave oven 151 (ie, the target object) and a background environment 152 .

[0266] For example, Fig.15 As shown, the left-eye image and the right-eye image may also include a real scene 153 where the virtual model is located.

[0267] The specific form of the virtual model and the specific form of the target surface can be found in the corresponding contents of the above embodiments, which will not be described in detail here.

[0268] 1402. In response to a background switching operation triggered by a user, a target background layer image of a target background environment and a target object layer image of the target object are acquired.

[0269] Optionally, in response to a background switching operation triggered by the user, a target background environment is determined. The determination process may refer to the corresponding contents in the above embodiments and will not be described in detail here.

[0270] Exemplarily, the lighting effect of the target object in the target object layer image is adapted to the lighting effect of the target background environment in the target background layer image.

[0271] Exemplarily, the target background layer image includes a target background environment.

[0272] Exemplarily, the target object is included in the target object layer image of the target object.

[0273] Exemplarily, the target object includes a first component and a second component, and the target object layer image of the target object includes a first target layer image of the first component and a second target layer image of the second component.

[0274] 1403. Determine a target composite image according to the target background layer image and the target object layer image.

[0275] Exemplarily, the target background layer image and the target object layer image are synthesized to obtain a target synthesized image.

[0276] Exemplarily, the target background layer image, the first target layer image of the first component, and the second target layer image of the second component are synthesized to obtain a target synthesized image.

[0277] 1404. Generate a transition image corresponding to the transition moment according to the background layer image, the target background layer image and the background animation configuration.

[0278] There are multiple transition moments, and the method for determining the multiple transition moments can refer to the corresponding contents in the above embodiments, which will not be described in detail here.

[0279] The generation process of the transition image can refer to the generation process of the above-mentioned transition left image, which will not be described in detail here.

[0280] 1405. At the transition moment, update the material texture image of the target surface according to the transition image corresponding to the transition moment.

[0281] At each transition moment, the material texture image of the target surface is updated according to the transition image corresponding to the transition moment.

[0282] Among them, different transition moments correspond to different transition images.

[0283] 1406. When the animation ends, update the material texture image of the target surface according to the target composite image.

[0284] Among them, the transition moment is earlier than the animation end moment.

[0285] Exemplarily, the monocular rendering shader writes the transition image to the texture area corresponding to the material texture of the target surface, thereby updating the material texture image of the target surface. In the embodiment of the present application, it is not necessary to distinguish between the left eye texture area and the right eye texture area.

[0286] Exemplarily, the monocular rendering shader writes the target synthetic image to the texture area corresponding to the material texture of the target surface, thereby updating the material texture image of the target surface. In the embodiment of the present application, it is not necessary to distinguish between the left eye texture area and the right eye texture area.

[0287] In the embodiment of the present application, the transition switching animation of the background environment in which the target object is located can be completed to enhance the user viewing experience.

[0288] In an optional implementation, the above step 1404 of “generating a transition image corresponding to a transition moment according to the background layer image, the target background layer image and the background animation configuration” can be implemented by the following steps:

[0289] S21, generating a transition background layer image corresponding to a transition moment according to the background layer image, the target background layer image and the background animation configuration.

[0290] The generation process of the transition background layer image can refer to the generation process of the transition left background layer image mentioned above, which will not be repeated here.

[0291] S22. Determine a transition image corresponding to the transition moment according to the transition background layer image corresponding to the transition moment.

[0292] In one embodiment, the transition background layer image corresponding to the transition moment may be used as the transition image corresponding to the transition moment. That is, the transition image does not contain the target object.

[0293] In another embodiment, a composite image of the transition background layer image corresponding to the transition moment and the target object layer image (or the object layer image of the target object) of the target object is used as the transition image corresponding to the transition moment. That is, the transition image includes the target object, and the target object has no animation switching effect, and only the background environment involves the animation switching effect.

[0294] In another embodiment, the above method may further include:

[0295] 1407. Generate a transition object layer image corresponding to the transition moment according to the object layer image, the target object layer image and the object animation configuration.

[0296] The generation process of the transition object layer image may refer to the generation process of the transition left object layer image in the above embodiment, which will not be described in detail here.

[0297] In the above S22, "determining the transition image corresponding to the transition moment according to the transition background layer image corresponding to the transition moment" may include: synthesizing the transition background layer image and the transition object layer image corresponding to the transition moment to obtain the transition image corresponding to the transition moment.

[0298] In this embodiment, the transition image includes a target object, and the target object involves an animation switching effect, and the background environment also involves an animation switching effect, and the animation switching effects involved in the two may be the same or different.

[0299] It should be noted here that: for the contents of each step not fully described in detail in the method provided in the embodiment of the present application, please refer to the corresponding contents in the above embodiment, which will not be repeated here. In addition, in addition to the above steps, the method provided in the embodiment of the present application may also include other parts or all of the steps in the above embodiments, which can be specifically referred to the corresponding contents in the above embodiments, which will not be repeated here.

[0300] Fig.16A flowchart of a display method for an XR device provided in an embodiment of the present application. The execution subject of the method may be an XR device. Exemplarily, the XR device includes: a left display screen corresponding to the left eye and a right display screen corresponding to the right eye. Exemplarily, an XR application is running on the XR device, and the XR device uses the XR application to implement the display method provided in an embodiment of the present application. Exemplarily, Fig.16 Each image involved in the corresponding embodiment is a monocular image. Fig.16 As shown, the method may include the following steps:

[0301] 1601. Display the target surface of the virtual model.

[0302] The left display screen displays a left-eye picture, and the right display screen displays a right-eye picture. The material texture image of the target surface in the left-eye picture is the same as the material texture image of the target surface in the right-eye picture.

[0303] Exemplarily, the left-eye image and the right-eye image may also include the real scene where the virtual model is located.

[0304] The specific form of the virtual model and the specific form of the target surface can be found in the corresponding contents of the above embodiments, which will not be described in detail here.

[0305] 1602. In response to an image switching operation triggered by a user, determine an exiting image and an entering image.

[0306] Exemplarily, the left-eye picture and the right-eye picture also display a control model corresponding to at least one alternative image.

[0307] Exemplarily, the material texture image of the surface of the control model corresponding to each candidate image is a thumbnail of the candidate image or a character pattern used to indicate the image name of the candidate image.

[0308] Exemplarily, the control model may be a simple 3D shape, such as a sphere or a cube, or a simple 2D plane, such as a circle or a rectangle.

[0309] Exemplarily, the image switching operation includes a trigger operation for a certain control model, such as a click operation, a double-click operation, etc., and the candidate image corresponding to the candidate control model is determined as the entry image.

[0310] Exemplarily, the current material texture image of the target surface may be determined as the exit image.

[0311] 1603. Generate a transition image corresponding to a transition moment according to the exit image, the entry image, and the animation configuration.

[0312] There are multiple transition moments, and the method for determining the multiple transition moments can refer to the corresponding contents in the above embodiments, which will not be described in detail here.

[0313] The generation process of the transition image can refer to the generation process of the above-mentioned transition left image, which will not be described in detail here.

[0314] 1604. At the transition moment, update the material texture image of the target surface according to the transition image corresponding to the transition moment.

[0315] 1605. When the animation ends, update the material texture image of the target surface according to the approach image.

[0316] The specific implementation of the above steps 1604 and 1605 can refer to the generation process of the above transition left image, which will not be described in detail here.

[0317] In the technical solution provided in the embodiment of the present application, the material texture image of the target surface of the virtual model can be switched in the form of a transition animation to enhance the user viewing experience.

[0318] It should be noted here that: for the contents of each step not fully described in detail in the method provided in the embodiment of the present application, please refer to the corresponding contents in the above embodiment, which will not be repeated here. In addition, in addition to the above steps, the method provided in the embodiment of the present application may also include other parts or all of the steps in the above embodiments, which can be specifically referred to the corresponding contents in the above embodiments, which will not be repeated here.

[0319] Fig.17 FIG. 1 is a schematic diagram showing the structure of an electronic device provided by an embodiment of the present application. Fig.17As shown, the electronic device includes a memory 1101, a processor 1102 and a display 1104. Among them, the display 1104 includes: a left display screen and a right display (not shown). The memory 1101 can be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method for operating on the electronic device. The memory 1101 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, SRAM), electrically erasable programmable read only memory (Electrically Erasable Programmable read only memory, EEPROM), erasable programmable read only memory (Electrical Programmable Read Only Memory, EPROM), programmable read only memory (Programmable Read Only Memory, PROM), read only memory (Read Only Memory, ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0320] The memory 1101 is used to store programs;

[0321] The display 1104 is used to display images;

[0322] The processor 1102 is coupled to the memory 1101 and the display 1104, and is used to execute the program stored in the memory 1101 to implement the methods provided by the above-mentioned method embodiments.

[0323] Further, if Fig.17 As shown, the electronic device also includes: a communication component 1103, a power component 1105, an audio component 1106 and other components. Fig.17 Only some components are shown schematically, which does not mean that the electronic device only includes Fig.17 Components shown.

[0324] in,

[0325] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a computer, the steps or functions of the methods provided in the above-mentioned method embodiments can be implemented.

[0326] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, can implement the steps or functions of the methods provided in the above-mentioned method embodiments.

[0327] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0328] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM (Read Only Memory) / RAM (Random Access Memory), a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0329] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0330] Having thus described several aspects and embodiments of the technology of the present application, it will be appreciated that various changes, modifications and improvements will be readily conceived by those of ordinary skill in the art. These changes, modifications and improvements are intended to fall within the spirit and scope of the technology described in this application. Therefore, it will be appreciated that the foregoing embodiments are presented only by way of example, and within the scope of the appended claims and their equivalents, embodiments of the present invention may be practiced in a manner different from that specifically described. In addition, if two or more features, systems, articles, materials and / or methods described herein are not mutually contradictory, any combination of these features, systems, articles, materials and / or methods is included within the scope of this disclosure.

[0331] Moreover, as mentioned above, some aspects can be embodied as one or more methods. The actions performed as part of the method can be ordered in any suitable manner. Therefore, embodiments can be constructed in which the actions are performed in a sequence different from the sequence shown, which can include performing some actions simultaneously, even if shown as sequential actions in the illustrative embodiments.

Claims

1. A display method, applied to the display of an extended reality device, wherein the extended reality device supports the display of binocular images, characterized in that: include: Display a left-eye picture, wherein the left-eye picture displays a target surface of a virtual model, wherein a material texture image of the target surface in the left-eye picture is determined by a left image, and the left image contains a target object; Display a right-eye picture, wherein the target surface is displayed in the right-eye picture, and a material texture image of the target surface in the right-eye picture is determined by a right image, and the right image contains the target object; The left image and the right image are used to form a three-dimensional image of the target object in human eyes.

2. The method according to claim 1, characterized in that Also includes: In response to a rotation operation triggered by a user on the target object, updating the material texture images of the target surface in the left-eye image and the right-eye image according to a first preset frequency; The orientation of the three-dimensional image is different before and after each update.

3. The method according to claim 2, characterized in that In response to a rotation operation triggered by a user on the target object, updating the material texture image of the target surface in the left-eye picture and the right-eye picture according to a first preset frequency, including: In response to the rotation operation, determining the target direction to which the three-dimensional image needs to rotate at every preset time interval, wherein the preset time interval is reciprocal to the first preset frequency; Determining a target image according to the target orientation; According to the target image, the material texture image of the target surface in the left-eye picture and the material texture image of the target surface in the right-eye picture are updated.

4. The method according to claim 3, characterized in that Updating the material texture image of the target surface in the left-eye picture and the material texture image of the target surface in the right-eye picture according to the target image, comprising: Determine a target left image and a target right image according to the target image, wherein the target left image includes the target object, the target right image includes the target object, and the target left image and the target right image are used to form a three-dimensional image of the target object in the eyes of a human being, with the orientation of the target object being the orientation of the target; According to the target left image, updating the material texture image of the target surface in the left eye picture; According to the target right image, the material texture image of the target surface in the right eye picture is updated.

5. The method according to claim 3, characterized in that: The rotation operation includes a drag operation; In response to the rotation operation, determining the target orientation to which the three-dimensional image needs to rotate at every preset time interval includes: In response to the drag operation, at every preset time interval, a target direction to which the three-dimensional image needs to be turned is determined according to the current operation point position of the drag operation.

6. The method according to claim 3, characterized in that The target object includes a first component and a second component, the target image includes a first target layer image and a second target layer image, the first target layer image includes the first component, and the second target layer image includes the second component.

7. The method according to claim 6, characterized in that Also includes: Preloading a plurality of first layer images of the first component and a plurality of second layer images of the second component into the memory of the extended reality device, wherein the plurality of first layer images correspond one-to-one to a plurality of preset orientations; and the plurality of second layer images correspond one-to-one to the plurality of preset orientations; Determining a target image according to the target orientation includes: Determine the first layer image corresponding to the target orientation as the first target layer image; The second layer image corresponding to the target orientation is determined as the second target layer image.

8. The method according to any one of claims 1 to 7, characterized in that The left image and the right image also include the background environment where the target object is located.

9. The method according to claim 8, characterized in that The left image and the right image are obtained by synthesizing the background layer image of the background environment and the object layer image of the target object.

10. The method according to claim 9, characterized in that Also includes: In response to a background switching operation triggered by a user, determining a target background environment; Acquire a target background layer image of the target background environment and a target object layer image of the target object; Determining a target composite image according to the target background layer image and the target object layer image; At a first moment, the material texture image of the target surface in the left-eye picture and the material texture image of the target in the right-eye picture are updated according to the target composite image.

11. The method according to claim 10, characterized in that Also includes: Determine a left background layer image and a right background layer image according to the background layer image; According to the target background layer image, determining a target left background layer image and a target right background layer image; Generate a transition left background layer image corresponding to a transition moment according to the left background layer image, the target left background layer image and the background animation configuration, wherein the transition moment is multiple; Generate a transition right background layer image corresponding to the transition moment according to the right background layer image, the target right background layer image and the background animation configuration; Determine a transition left image and a transition right image corresponding to the transition moment according to the transition left background layer image and the transition right background layer image corresponding to the transition moment respectively; At the transition moment, according to the transition left image corresponding to the transition moment, the material texture image of the target surface in the left eye picture is updated, and according to the transition right image corresponding to the transition moment, the material texture image of the target surface in the right eye picture is updated; The transition moment is earlier than the first moment.

12. The method according to claim 11, characterized in that Also includes: Determine a left object layer image and a right object layer image according to the object layer image; Determine a target left object layer image and a target right object layer image according to the target object layer image; Generate a transition left object layer image corresponding to the transition moment according to the left object layer image, the target left object layer image and the object animation configuration; generating a transition right object layer image corresponding to the transition moment according to the right object layer image, the target right object layer image and the object animation configuration; Determining the transition left image and the transition right image corresponding to the transition moment according to the transition left background layer image and the transition right background layer image corresponding to the transition moment, respectively, comprises: Synthesizing the transition left background layer image and the transition left object layer image corresponding to the transition moment to obtain a transition left image corresponding to the transition moment; The transition right background layer image and the transition right object layer image corresponding to the transition moment are synthesized to obtain a transition right image corresponding to the transition moment.

13. The method according to claim 11, characterized in that Also includes: Get the preset animation duration; According to a preset animation duration and a second preset frequency, a plurality of transition moments are determined.

14. A display method, applied to the display of an extended reality device, characterized in that: include: Displaying a target surface of a virtual model, wherein a material texture image of the target surface is determined by an image, wherein the image is obtained by synthesizing a background layer image of a background environment and an object layer image of the target object; In response to a background switching operation triggered by a user, acquiring a target background layer image of a target background environment and a target object layer image of the target object; Determining a target composite image according to the target background layer image and the target object layer image; Generate a transition image corresponding to a transition moment according to the background layer image, the target background layer image and the background animation configuration, wherein the transition moment is multiple; At the transition moment, updating the material texture image of the target surface according to the transition image corresponding to the transition moment; When the animation ends, the material texture image of the target surface is updated according to the target composite image.

15. The method according to claim 14, characterized in that Generating a transition image corresponding to a transition moment according to the background layer image, the target background layer image and the background animation configuration, including: Generate a transition background layer image corresponding to the transition moment according to the background layer image, the target background layer image and the background animation configuration; According to the transition background layer image corresponding to the transition moment, a transition image corresponding to the transition moment is determined.

16. The method according to claim 15, characterized in that Also includes: Generate a transition object layer image corresponding to the transition moment according to the object layer image, the target object layer image and the object animation configuration; Determining a transition image corresponding to the transition moment according to the transition background layer image corresponding to the transition moment includes: The transition background layer image and the transition object layer image corresponding to the transition moment are synthesized to obtain a transition image corresponding to the transition moment.

17. A display method, applied to the display of an extended reality device, characterized in that: include: Display the target surface of the virtual model; In response to an image switching operation triggered by a user, determining an exiting image and an entering image; Generate a transition image corresponding to a transition moment according to the exit image, the entry image and the animation configuration, wherein the transition moment is multiple; At the transition moment, updating the material texture image of the target surface according to the transition image corresponding to the transition moment; When the animation ends, the material texture image of the target surface is updated according to the approach image.

18. An extended reality device, characterized in that: include: Memory, processor and display wherein: The memory is used to store programs; The display is used to display images; The processor is coupled to the memory and the display, and is configured to execute the program stored in the memory to implement the method according to any one of claims 1 to 17.

19. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a computer, the method according to any one of claims 1 to 17 can be implemented.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 17 is implemented.

Images