Image processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides an image processing method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring first image information, wherein the first image information is used for generating a first image displayed by augmented reality equipment; obtaining first depth information matched with the first image information, wherein the first depth information is obtained based on depth information of a real scene image matched with the predicted display time of the first image; and performing synthesis rendering based on the first image information and the first depth information to obtain the first image. According to the image processing method and device, the electronic equipment, the storage medium and the program product, the display effect of the composite image can be improved, and the user experience is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an image processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Extended Reality (XR) refers to the use of computers to combine the real and virtual worlds, creating an interactive virtual environment. XR is also a collective term for various technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). By integrating the visual interaction technologies of these three technologies, it provides users with an immersive experience that seamlessly transitions between the virtual and real worlds.

[0003] Interaction between the virtual and real worlds typically requires depth information. However, in related technologies, depth information is generated and scene images are rendered separately, resulting in a mismatch between the two and poor display quality, thus reducing the user experience. Summary of the Invention

[0004] In view of this, the purpose of this disclosure is to provide an image processing method, apparatus, electronic device, storage medium, and program product.

[0005] To achieve the above objectives, the first aspect of this disclosure provides an image processing method, comprising: Acquire first image information, which is used to generate a first image displayed by the extended reality device; Obtain first depth information that matches the first image information, wherein the first depth information is obtained based on the depth information of a real scene image that matches the expected display time of the first image; The first image is obtained by synthesizing and rendering based on the first image information and the first depth information.

[0006] In some embodiments, the first image includes at least one of a virtual object and a real object, wherein the real object is determined based on the real-world scene image; the synthetic rendering based on the first image information and the first depth information includes at least one of the following: Determine the occlusion relationship information between the virtual object and the real object, and display the virtual object and the real object in the first image based on the occlusion relationship information; Determine the relative positional relationship between the virtual object and the real object, and display the interaction information between the virtual object and the real object in the first image based on the relative positional relationship; Determine the light and shadow information between the virtual object and the real object, and display the light and shadow imaging information between the virtual object and the real object in the first image based on the light and shadow information; Determine the spatial positioning information between the virtual object and the real object, and locate the virtual object and the real object displayed in the first image based on the spatial positioning information.

[0007] In some embodiments, the method further includes: During the process of compositing and rendering the second image, the first depth information is simultaneously predicted and obtained; Wherein, the second image is the frame preceding the first image, or the second image is an image preceding the first image and separated by at least one frame.

[0008] In some embodiments, the synchronous prediction for obtaining the first depth information includes: Obtain the estimated display time of the first image; The device pose of the extended reality device is predicted to obtain a first device pose that matches the expected display time of the first image; The real-world scene image is acquired, and depth information matching the first image is obtained based on the real-world scene image and the first device pose, thus obtaining the first depth information.

[0009] In some embodiments, obtaining the estimated display time of the first image includes: The estimated display time of the second image is obtained, and the estimated display time of the second image is determined based on the actual display time of the previous frame of the second image and the device frame rate of the extended reality device; Obtain the device frame rate of the extended reality device; The estimated display time of the first image is determined based on the estimated display time of the second image and the device frame rate.

[0010] In some embodiments, obtaining depth information matching the first image based on the real-world scene image and the first device pose includes: The real-world scene image and the first device posture are input into the first model, and the first depth information is obtained based on the output information of the first model. The first model is used to reproject the real-world scene image according to the first device posture to generate a predicted real-world scene image that matches the expected display time of the first image, obtain the depth information of the predicted real-world scene image, and determine the depth information of the predicted real-world scene image as the first depth information.

[0011] In some embodiments, the method wherein The extended reality device displays an image sequence according to the device frame rate of the extended reality device, and the image sequence includes the second image and the first image; The first model generates a set of depth information that matches the image sequence according to the device frame rate of the extended reality device, and the set of depth information includes the first depth information.

[0012] In some embodiments, the method wherein The first model generates at least one depth information from the depth information set based on the same real-world scene image; The at least one depth piece of information includes: The first depth information; And / or, The first depth information, and at least one of the following: at least one depth information matching at least one frame of image preceding the first image, and at least one depth information matching at least one frame of image following the first image.

[0013] In some embodiments, the method further includes: The first image is obtained by performing composite rendering based on the first image information and the first depth information using a graphics processor.

[0014] In some embodiments, the method further includes: During the process of synthesizing and rendering based on the first image information and the first depth information, the second depth information matching the third image is simultaneously predicted and obtained. The third image is either the frame following the first image, or the third image is an image following the first image with an interval of at least one frame.

[0015] A second aspect of this disclosure provides an image processing apparatus, comprising: The first acquisition module is configured to: acquire first image information, wherein the first image information is used to generate a first image displayed by the extended reality device; The second acquisition module is configured to: acquire first depth information that matches the first image information, wherein the first depth information is obtained based on the depth information of a real scene image that matches the expected display time of the first image; The compositing module is configured to perform compositing rendering based on the first image information and the first depth information to obtain the first image.

[0016] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image processing method as described in the first aspect.

[0017] In some embodiments, the electronic device includes a wearable device.

[0018] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the image processing method described in the first aspect.

[0019] The fifth aspect of this disclosure provides a computer program product including computer program instructions that, when executed on a computer, cause the computer to perform the image processing method as described in the first aspect.

[0020] As can be seen from the above, the image processing method, apparatus, electronic device, storage medium, and program product provided in this disclosure obtain first depth information by predicting the depth information of the real scene image that matches the expected display time of the first image. This allows the first image to be obtained before rendering and compositing, thus directly compositing and rendering the first image and the first depth information without blocking the image rendering process, without reducing the display frame rate of the wearable device. Furthermore, since the first depth information is the depth information of the real scene image that matches the expected display time of the first image, the relative relationship between the virtual object and the real object in the real scene is more closely matched at the expected display time of the first image, thereby improving the display effect of the synthesized image and enhancing the user experience. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1A A schematic diagram of an exemplary system provided by an embodiment of this disclosure is shown.

[0023] Figure 1B A schematic diagram of an exemplary image processing procedure is shown.

[0024] Figure 2A flowchart of an exemplary method is shown.

[0025] Figure 3 A flowchart illustrating an exemplary method provided by an embodiment of this disclosure is shown.

[0026] Figure 4 A schematic diagram of an exemplary image processing procedure provided by an embodiment of this disclosure is shown.

[0027] Figure 5 A schematic diagram of another exemplary image processing procedure provided by an embodiment of this disclosure is shown.

[0028] Figure 6 A schematic diagram of yet another exemplary image processing procedure provided by an embodiment of the present disclosure is shown.

[0029] Figure 7 A schematic diagram of an exemplary apparatus provided by an embodiment of the present disclosure is shown.

[0030] Figure 8 A schematic diagram of the hardware structure of an exemplary computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0032] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar words used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0033] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0034] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as electronic devices, applications, servers, or storage media performing the operations of this disclosed technical solution.

[0035] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0036] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0037] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0038] Figure 1A A schematic diagram of an exemplary extended reality system 100 provided in an embodiment of this disclosure is shown.

[0039] Extended Reality (XR) refers to the use of computers to combine the real and virtual worlds, creating a virtual environment that allows for human-computer interaction. XR technology can further include Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR), utilizing hardware devices and various technologies to merge virtual content with real-world scenes.

[0040] like Figure 1A As shown, the system 100 may include various types of extended reality devices, such as wearable devices, for example, head-mounted wearable devices (e.g., VR / AR glasses or head-mounted displays (HMDs)) 104, control handles 108, etc. In some scenarios, a camera / camera 110 may also be provided for taking photos of the operator (user) 102. In some embodiments, when the aforementioned devices do not have processing functions, the system 100 may also include an external control device 112 for providing processing functions. The control device 112 may be, for example, a mobile phone, computer, or other computer device. In some embodiments, when any of the aforementioned devices acts as a control device or a main control device, it can interact with other devices in the system 100 through wired or wireless communication methods to achieve information exchange.

[0041] In system 100, user 102 can interact with extended reality system 100 using head-mounted wearable device 104 and control handle 108. In some scenarios, system 100 can use images captured by camera / video camera 110 to recognize user 102's posture, gestures, etc., and then complete the interaction with user 102 based on the recognized posture and gestures. In some embodiments, user 130 can also perform gesture input using bare hands. Head-mounted wearable device 104 can capture images in front of it in real time using a camera or other camera positioned in front of the head-mounted wearable device 104, and recognize user 130's gestures by recognizing these images.

[0042] In some embodiments, such as Figure 1A As shown, system 100 can also communicate with server 114 and obtain data from server 114, such as images, audio, and video, and can output this data through head-mounted wearable device 104, for example, displaying images or videos on the display screen of head-mounted wearable device 104, playing audio and video audio through the speaker of head-mounted wearable device 104, etc. In some embodiments, such as Figure 1A As shown, server 114 can retrieve the required data, such as images, audio, and video, from database server 116, which is used to store data.

[0043] In some embodiments, a data acquisition unit for collecting information may be provided on the head-mounted wearable device 104. The type of data acquisition unit can be varied.

[0044] In some embodiments, the acquisition unit may further include an environment acquisition unit and a positioning and tracking unit. The environment acquisition unit can be used to acquire environmental information around (e.g., in front of) the wearable device 104, and the positioning and tracking unit can be used to perform positioning and tracking of the wearable device 104. Optionally, the environment acquisition unit may include, but is not limited to, photosensitive elements such as a three-color camera (e.g., an RGB camera), a depth camera, a binocular camera, or a laser. The positioning and tracking unit may include, but is not limited to, modules such as visual real-time localization and mapping (visual SLAM), an inertial measurement unit (IMU), a global positioning system (GPS), ultra-wideband wireless communication technology (UWB), and lasers.

[0045] In some embodiments, the head-mounted wearable device 104 may also be equipped with a velocity sensor, acceleration sensor, angular velocity sensor (e.g., gyroscope), etc., for collecting velocity or acceleration information of the head-mounted wearable device 104. Similarly, the operating handle 108 may also be equipped with a velocity sensor, acceleration sensor, angular velocity sensor (e.g., gyroscope), etc., for collecting velocity or acceleration information of the worn glove 106. It should be noted that, in addition to being mounted on the head-mounted wearable device 104 and the operating handle 108, the aforementioned data collection units may also be directly attached to the body parts of the user 102 without relying on hardware devices, thereby collecting relevant information of that body part, such as velocity, acceleration, or angular velocity information, or information collected by other sensors or data collection units.

[0046] In some embodiments, the head-mounted wearable device 104 may also be equipped with a camera or webcam for taking photos of the operator (user) 102 (e.g., photos of hands or feet) and environmental images.

[0047] In some embodiments, the system 100 can identify the posture, gestures, etc. of the user 102 by collecting information, and then perform corresponding interactions based on the identified user posture and gestures.

[0048] Figure 1B A schematic diagram of an exemplary head-mounted wearable device 104 is shown.

[0049] like Figure 1B As shown, the head-mounted wearable device 104 may include a lens barrel 1042, which may house a display screen 1044 for displaying images and an optical component 1046 for processing light. Optionally, the optical component 1046 may further include multiple lenses (e.g., lenses 1046A and 1046B). The combination of multiple lenses can project the light emitted from the display screen 1044 into the human eye 1022, allowing the human eye 1022 to view the image displayed on the display screen 1044. It is understood that... Figure 1B The diagram only shows a single-sided structure of the head-mounted wearable device 104 as an example. In order to achieve binocular display, the head-mounted wearable device 104 may include two lens barrel structures arranged side by side.

[0050] In some embodiments, such as Figure 1B As shown, the head-mounted wearable device 104 may also be equipped with a camera 1048 for capturing images of the human eye. The camera 1048 may be a charge-coupled device (CCD) image sensor, a complementary metal-oxide-semiconductor (CMOS) image sensor, etc.

[0051] Optionally, the camera 1048 can be an eye-tracking (ET) camera, and the human eye images it acquires can be used to achieve functions such as pupil distance estimation and eye-tracking.

[0052] In some embodiments, depth information is obtained by acquiring environmental information around the wearable device 104 through the environment acquisition unit, and then combined with an image containing virtual objects generated by the wearable device or control device 112, thereby enabling interaction between virtual objects and the real environment. The depth information can be, for example, a depth map.

[0053] As described in the background section, in related technologies, such as Figure 2 As shown, depth information is generated and scene images are rendered through different processes, and the two are not matched.

[0054] The inventors of this disclosure discovered that the generation of depth information does not match the rendering of scene images because the rendering speed of scene images is relatively fast, usually greater than or equal to 72 Hz; while the generation speed of depth information is relatively slow. Due to hardware limitations, the generation of depth information is usually less than or equal to 30 Hz, which leads to the mismatch between the generation of depth information and the rendering of scene images.

[0055] In related technologies, the following two methods are typically used to synthesize depth information and scene images: (1) The latest depth information and scene images are synthesized and displayed on the display screen of the wearable device 104.

[0056] like Figure 2 As shown, image A is synthesized with depth map a; for image B, the latest depth information is still depth map a, so image B is also synthesized with depth map a.

[0057] In this case, for image A and image B, the position and angle of user 102 relative to the real scene may have changed, but the depth information is the same. This causes the relative relationship between the virtual object determined based on the depth information and the real object in the real scene to not match when the depth map a is synthesized from image A and image B respectively, resulting in a poor display effect of the synthesized image.

[0058] (2) Reduce the display frequency of scene images to match the generation frequency of depth information.

[0059] like Figure 2 As shown, for image B, after the depth map b is generated, image B is combined with the depth map b and then displayed. This causes the frame rate of the displayed image of the wearable device 104 to decrease, resulting in a decrease in display effect.

[0060] In view of this, the present disclosure provides an image processing method that can solve or partially solve the above problems to a certain extent.

[0061] like Figure 3 As shown, the image processing method includes: Step S101: Obtain first image information, which is used to generate the first image displayed by the extended reality device.

[0062] In this embodiment, the first image information is the image information used to generate the first image displayed by the extended reality device.

[0063] The first image information may include basic information such as geometric information, surface material information, lighting information, and environmental information used to generate the first image, and may also include information such as eye-tracking data and channel rendering parameters used to generate extended reality display images. This embodiment does not limit this.

[0064] In some embodiments, the first image information may include image information for generating virtual objects, or it may include image information for generating real objects in a real scene. This embodiment does not limit this.

[0065] Step S103: Obtain first depth information that matches the first image information. The first depth information is obtained based on the depth information of a real-world scene image that matches the expected display time of the first image.

[0066] In this embodiment, the display time of the first image can be predicted first to determine the expected display time of the first image; then, the first depth information of the real scene image that matches the expected display time can be obtained. In some embodiments, the first depth information matching the first image information can be obtained by predicting the depth information of the real scene image that matches the expected display time.

[0067] like Figure 4 , Figure 6 As shown, taking image information B as the first image information, image B as the first image, depth map b as the first depth information, and image a as the real scene image as the real scene image, the expected display time t2 of image B is first determined; then, the depth information of the real scene image at the expected display time t2 is predicted based on the obtained real scene image a to obtain depth map b, and this depth map b needs to be obtained before the first image is synthesized and rendered based on the first image information.

[0068] Since the first depth information is predicted in advance before the first image is synthesized and rendered, the image rendering process can be directly synthesized and rendered by combining the first depth information after obtaining the first image information without blocking the first image rendering process, so as not to reduce the display frame rate of the wearable device.

[0069] Since the first depth information is the depth information of the real scene image that matches the expected display time of the first image, the relative relationship between the virtual object and the real object in the real scene is more matched at the expected display time of the first image determined based on the first depth information, thereby improving the display effect of the synthesized image.

[0070] In this embodiment, before rendering the first image, first depth information matching the first image information of the first image is obtained in advance. The first depth information is obtained by predicting the depth information of the real scene image that matches the expected display time of the first image, thereby predicting the depth information of the real scene image that matches the first image after it is generated and displayed.

[0071] Step S105: Perform synthetic rendering based on the first image information and the first depth information to obtain the first image.

[0072] The first image information and the depth information of the real scene image that matches the first image after it is generated and displayed are synthesized and rendered to obtain the first image, which is then displayed on the display screen of the wearable device.

[0073] In this embodiment, the first depth information is obtained by predicting the depth information of the real-world scene image that matches the expected display time of the first image. This allows the first image to be rendered and synthesized before the first image information is rendered, thus avoiding blocking the image rendering process and enabling direct synthesis and rendering of the first image and the first depth information to obtain the first image without reducing the display frame rate of the wearable device. Furthermore, since the first depth information is the depth information of the real-world scene image that matches the expected display time of the first image, the relative relationship between the virtual object and the real object in the real-world scene is more accurately matched at the expected display time of the first image, thereby improving the display effect of the synthesized image and enhancing the user experience.

[0074] In some embodiments, the first image includes at least one of a virtual object and a real object, wherein the real object is determined based on the real-world scene image. The virtual object may be a digital entity generated by a wearable device or control device 112 using graphics rendering technology; the real object may be an entity object acquired by the environment acquisition unit of the wearable device and displayed on the display screen of the wearable device, or an object acquired by the environment acquisition unit of the wearable device and then generated using graphics rendering technology. This embodiment does not limit the specific object to this embodiment.

[0075] The composite rendering based on the first image information and the first depth information in step S105 includes: determining the occlusion relationship information between the virtual object and the real object, and displaying the virtual object and the real object in the first image based on the occlusion relationship information.

[0076] In this embodiment, the occlusion relationship information between the virtual object and the real object can be determined based on the first image information and the corresponding first depth information, and the occlusion relationship between the virtual object and the real object can be displayed in the first image based on the occlusion relationship information.

[0077] For example, a virtual cat can correctly disappear behind a real cabinet; a virtual character can walk in front of a real table and block part of it.

[0078] The composite rendering based on the first image information and the first depth information in step S105 includes: determining the relative positional relationship between the virtual object and the real object, and displaying the interaction information between the virtual object and the real object in the first image based on the relative positional relationship.

[0079] In this embodiment, the relative positional relationship between the virtual object and the real object can be determined based on the first image information and the corresponding first depth information, and then the boundary of the real object can be determined based on the relative positional relationship, so that the virtual object can interact with the real object.

[0080] For example, a virtual cup can be placed on a real table instead of floating in the air; a virtual ball can be thrown at a real wall and bounce back; virtual objects can slide on a real slope, etc.

[0081] The composite rendering based on the first image information and the first depth information in step S105 includes: determining the light and shadow information between the virtual object and the real object, and displaying the light and shadow imaging information between the virtual object and the real object in the first image based on the light and shadow information.

[0082] In this embodiment, the light and shadow information between the virtual object and the real object can be determined based on the first image information and the corresponding first depth information. The light and shadow information may include, for example, illumination and projection, so as to realize the light and shadow imaging information between the virtual object and the real object based on the illumination and projection between the two.

[0083] For example, virtual objects can cast shadows onto the real ground, and real objects can cast shadows onto virtual objects; the surface of a virtual object can receive shadows generated by the real scene, making its brightness changes consistent with the real scene, etc.

[0084] The composite rendering based on the first image information and the first depth information in step S105 includes: determining the spatial positioning information between the virtual object and the real object, and locating the virtual object and the real object displayed in the first image based on the spatial positioning information.

[0085] In this embodiment, the spatial positioning information between the virtual object and the real object can be determined based on the first image information and the corresponding first depth information. The spatial positioning information can be, for example, the user's position and orientation in a certain space, so that the virtual object and the real object can be located based on the spatial positioning information.

[0086] For example, a virtual picture frame can be fixed to a real wall, and even if the user leaves the room and returns, the virtual picture frame will remain fixed to the original wall and will not drift; a virtual window can be fixed to the wall and will not shake due to the slight movement of the user's head.

[0087] In some embodiments, the method further includes: synchronously predicting and obtaining the first depth information during the process of compositing and rendering the second image; wherein the second image is the previous frame of the first image, or the second image is an image that precedes the first image and is separated by at least one frame.

[0088] In this embodiment, the second image is the frame preceding the first image. For example, as shown... Figure 4 As shown, if the first image is image B, then the second image can be image A; if the first image is image C, then the second image can be image B; if the first image is image D, then the second image can be image C.

[0089] In this embodiment, the second image can also be an image that precedes the first image and is separated by at least one frame. For example, such as... Figure 4 As shown, if the first image is image C, then the second image can be image A; if the first image is image D, then the second image can be image B or image A. This embodiment does not limit this.

[0090] In this embodiment, when performing composite rendering on the previous frame or multiple frames of the first image, the first depth information matching the first image is simultaneously predicted and generated. Thus, when the first image needs to be rendered based on the first image information, the first depth information matching the first image information has already been generated. After obtaining the first image information, the image rendering process can directly perform composite rendering based on the first image information and the first depth information to obtain the first image, thereby avoiding blocking the image rendering process and not reducing the display frame rate of the wearable device's display image.

[0091] In this embodiment, when the second image is an image that precedes the first image and is at least one frame apart, more time can be reserved for the generation of the first depth information, thereby ensuring that the first depth information matching the first image information can be obtained before rendering the first image information. When the second image is the image preceding the first image, the real scene image obtained at this time is closer to the expected display time of the first image. Therefore, the depth information of the real scene image that matches the expected display time of the first image obtained by prediction, i.e., the first depth information, is more accurate, thereby improving the display effect of the first image.

[0092] In some embodiments, the synchronous prediction for obtaining the first depth information includes: Step S201: Obtain the estimated display time of the first image.

[0093] In this embodiment, the display time of the first image can be predicted first, thereby determining the expected display time of the first image.

[0094] The step S201, which involves obtaining the estimated display time of the first image, includes: Step S2011: Obtain the estimated display time of the second image. The estimated display time of the second image is determined based on the actual display time of the previous frame of the second image and the device frame rate of the extended reality device.

[0095] The actual display time of the previous frame of the second image can be determined based on the display time of the previous frame of the already displayed second image.

[0096] The device frame rate of the extended reality device can determine the time required to render and generate each frame of the image. Thus, the expected display time of the second image can be predicted based on the actual display time of the previous frame of the second image and the time required to render and generate the second image.

[0097] Step S2013: Obtain the device frame rate of the extended reality device.

[0098] In this embodiment, the time required to render and generate each frame of image can be determined based on the device frame rate of the extended reality device, thus obtaining the time required to render and generate the first image.

[0099] Step S2015: Determine the expected display time of the first image based on the expected display time of the second image and the device frame rate.

[0100] In this embodiment, the expected display time of the first image can be predicted based on the expected display time of the second image and the time required to render and generate the first image.

[0101] like Figure 5 As shown, taking image B as the first image, the estimated display time t2 of the first image can be predicted.

[0102] Step S203: Predict the device pose of the extended reality device to obtain a first device pose that matches the expected display time of the first image.

[0103] In this embodiment, the device posture of the extended reality device at the expected display time of the first image can be predicted based on the historical device posture information of the extended reality device, so as to obtain the first device posture.

[0104] The device posture can include the position and orientation of the extended reality device. In this embodiment, the motion state of the extended reality device, such as speed, acceleration, and angular velocity, can be estimated based on historical device posture information. This allows for the calculation of the device posture at a future point in time, corresponding to the expected display time of the first image.

[0105] like Figure 5 As shown, device pose b is the first device pose that matches image B at the expected display time t2 of the first image.

[0106] Step S205: Obtain the real-world scene image based on the real-world scene, and obtain depth information matching the first image based on the real-world scene image and the first device pose to obtain the first depth information.

[0107] In this embodiment, the latest real-world scene image is obtained, such as... Figure 5 As shown, the latest real-world scene image is real-world scene image a. Then, based on real-world scene image a and device pose b, the first depth information, i.e., depth map b, matching image B is obtained. This allows the first depth information, i.e., depth map b, to be synthesized and rendered with the first image information, i.e., image information b, to obtain the first image, i.e., image B.

[0108] like Figure 5 As shown, when the first image is image C, during the process of compositing and rendering image B, the expected display time t3 of the first image can be predicted, and the device posture at the expected display time t3 of the first image can be predicted to obtain the device posture c; the latest real scene image, i.e., real scene image a, is obtained; then, based on the real scene image a and the device posture c, a depth map c matching image C is obtained, so that the depth map c and the image information of image C can be composited and rendered to obtain image C.

[0109] In some embodiments, step S205, which involves obtaining depth information matching the first image based on the real-world scene image and the first device pose, includes: inputting the real-world scene image and the first device pose into a first model, and obtaining the first depth information based on the output information of the first model; wherein the first model is used to reproject the real-world scene image according to the first device pose to generate a predicted real-world scene image that matches the expected display time of the first image, obtain the depth information of the predicted real-world scene image, and determine the depth information of the predicted real-world scene image as the first depth information.

[0110] In this embodiment, the first depth information can be generated using a first model. Specifically, a real-world scene image and the first device pose can be used as inputs to the first model, and the first depth information is generated based on the output of the first model.

[0111] The first model can be a deep model, or it can be other types of artificial intelligence models. This embodiment does not limit this.

[0112] In this embodiment, the first model can reproject the latest real scene image according to the first device posture that matches the expected display time of the first image, thereby obtaining a predicted real scene image that matches the expected display time of the first image, and obtain the depth information of the predicted real scene image as the first depth information.

[0113] In this embodiment, a first device pose is obtained by predicting the device pose of the extended reality device. The first device pose is matched with the expected display time of the first image. In this way, even if the latest real scene image does not match the expected display time of the first image, the depth information that matches the expected display time of the first image can still be obtained by reprojecting the latest real scene image using the first device pose.

[0114] In some embodiments, the extended reality device displays an image sequence according to the device frame rate of the extended reality device, the image sequence including the second image and the first image; and the first model generates a depth information set matching the image sequence according to the device frame rate of the extended reality device, the depth information set including the first depth information.

[0115] In this embodiment, as Figure 5 , Figure 6 As shown, the extended reality device displays an image sequence including a second image and a first image according to the device frame rate of this application. The image sequence may specifically include: ...image A, image B, image C, image D...

[0116] During the rendering and compositing of image A, the device pose of the extended reality device is predicted to set the device pose b. The first model then reprojects the newly acquired real-world scene image a based on the set device pose b to obtain the depth map b used to generate image B.

[0117] like Figure 6 As shown, after image A is generated, the image rendering process acquires the image information B of image B and performs composite rendering based on the generated depth map b to obtain image B. During the composite rendering of image information B and depth map b, the device pose of the extended reality device is predicted to set the device pose c. The first model then reprojects the newly acquired real scene image a based on the set device pose c to obtain the depth map c used to generate image C.

[0118] After image B is generated, the image rendering process acquires the image information of image C and performs composite rendering based on the generated depth map c to obtain image C. During the composite rendering of the image information of image C and the depth map c, the device pose of the extended reality device is predicted to set the device pose d. The first model then reprojects the newly acquired real scene image b based on the set device pose d to obtain the depth map d used to generate image D.

[0119] Repeat the above steps to achieve image display on the extended reality device.

[0120] In this embodiment, the extended reality device displays the image sequence according to the device frame rate of this application; at the same time, the first model generates a set of depth information matching the image sequence according to the device frame rate of the extended reality device, so that the first image information and the first depth information can be directly synthesized and rendered to obtain the first image without blocking the image rendering process, and the display frame rate of the wearable device will not be reduced.

[0121] In some embodiments, the first model generates at least one depth information from the depth information set based on the same real-world scene image. The at least one depth information includes: the first depth information; and / or, the first depth information, and at least one of the following: at least one depth information matched with at least one frame of an image preceding the first image, and at least one depth information matched with at least one frame of an image following the first image.

[0122] In this embodiment, due to hardware performance limitations, the generation frequency of real-world scene images is lower than the device frame rate of the extended reality device. Therefore, one or more depth information pieces will be generated based on the same real-world scene image. Specifically, the same real-world scene image may be used to generate only one depth information piece, namely the first depth information; or the same real-world scene image may be used to generate multiple depth information pieces, for example, the first depth information and at least one depth information piece matched with at least one frame preceding the first image, the first depth information and at least one depth information piece matched with at least one frame following the first image, or the first depth information, at least one depth information piece matched with at least one frame preceding the first image, and at least one depth information piece matched with at least one frame following the first image. This embodiment does not limit this specific approach.

[0123] In this embodiment, for real-world scene images captured by the same extended reality device, the amount of depth information generated based on different real-world scene images may be the same, or the amount of depth information generated based on different real-world scene images may be different. This embodiment does not limit this.

[0124] In some embodiments, the method further includes: performing composite rendering by a graphics processor based on the first image information and the first depth information to obtain the first image.

[0125] In this embodiment, since the first depth information is obtained before rendering and compositing the first image information, the first image can be directly synthesized and rendered by the graphics processing unit (GPU) to obtain the first image without occupying the computing resources of the central processing unit (CPU), thereby saving the computing resources of the central processing unit and improving the image processing efficiency of the extended reality device.

[0126] In some embodiments, the method further includes: during the process of synthesizing and rendering based on the first image information and the first depth information, simultaneously predicting and obtaining second depth information matching the third image; wherein the third image is the next frame image after the first image, or the third image is an image that follows the first image and is at least one frame apart.

[0127] In this embodiment, the third image is the frame following the first image. For example, as... Figure 4 As shown, if the first image is image A, then the third image can be image B; if the first image is image B, then the third image can be image C; if the first image is image C, then the third image can be image D.

[0128] In this embodiment, the third image can also be an image that follows the first image and is separated by at least one frame. For example, such as... Figure 4 As shown, if the first image is image A, then the third image can be image C or image D; if the first image is image B, then the third image can be image D. This embodiment does not limit this.

[0129] In this embodiment, when the first image is rendered, second depth information matching the third image is predicted and generated simultaneously. This way, when the third image needs to be rendered based on the image information of the third image, the second depth information matching the image information of the third image has already been generated. After obtaining the image information of the third image, the image rendering process can directly perform composite rendering based on the image information of the third image and the second depth information to obtain the third image, thus without blocking the image rendering process and without reducing the display frame rate of the wearable device's display image.

[0130] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0131] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0132] Based on the same inventive concept, corresponding to any of the above-described embodiments, this disclosure also provides an image processing apparatus.

[0133] refer to Figure 7 The device includes: The first acquisition module 11 is configured to: acquire first image information, the first image information being used to generate a first image displayed by the extended reality device; The second acquisition module 13 is configured to: acquire first depth information that matches the first image information, wherein the first depth information is obtained based on the depth information of a real scene image that matches the expected display time of the first image; The compositing module 15 is configured to perform compositing rendering based on the first image information and the first depth information to obtain the first image.

[0134] In some embodiments, the first image includes at least one of a virtual object and a real object, wherein the real object is determined based on the real-world scene image; the compositing module 15 is configured to implement at least one of the following: Determine the occlusion relationship information between the virtual object and the real object, and display the virtual object and the real object in the first image based on the occlusion relationship information; Determine the relative positional relationship between the virtual object and the real object, and display the interaction information between the virtual object and the real object in the first image based on the relative positional relationship; Determine the light and shadow information between the virtual object and the real object, and display the light and shadow imaging information between the virtual object and the real object in the first image based on the light and shadow information; Determine the spatial positioning information between the virtual object and the real object, and locate the virtual object and the real object displayed in the first image based on the spatial positioning information.

[0135] In some embodiments, the device is further configured to: During the process of compositing and rendering the second image, the first depth information is simultaneously predicted and obtained; Wherein, the second image is the frame preceding the first image, or the second image is an image preceding the first image and separated by at least one frame.

[0136] In some embodiments, the synchronous prediction for obtaining the first depth information includes: Obtain the estimated display time of the first image; The device pose of the extended reality device is predicted to obtain a first device pose that matches the expected display time of the first image; The real-world scene image is acquired, and depth information matching the first image is obtained based on the real-world scene image and the first device pose, thus obtaining the first depth information.

[0137] In some embodiments, obtaining the estimated display time of the first image includes: The estimated display time of the second image is obtained, and the estimated display time of the second image is determined based on the actual display time of the previous frame of the second image and the device frame rate of the extended reality device; Obtain the device frame rate of the extended reality device; The estimated display time of the first image is determined based on the estimated display time of the second image and the device frame rate.

[0138] In some embodiments, obtaining depth information matching the first image based on the real-world scene image and the first device pose includes: The real-world scene image and the first device posture are input into the first model, and the first depth information is obtained based on the output information of the first model. The first model is used to reproject the real-world scene image according to the first device posture to generate a predicted real-world scene image that matches the expected display time of the first image, obtain the depth information of the predicted real-world scene image, and determine the depth information of the predicted real-world scene image as the first depth information.

[0139] In some embodiments, the device is further configured to: The extended reality device displays an image sequence according to the device frame rate of the extended reality device, and the image sequence includes the second image and the first image; The first model generates a set of depth information that matches the image sequence according to the device frame rate of the extended reality device, and the set of depth information includes the first depth information.

[0140] In some embodiments, the device is further configured to: The first model generates at least one depth information from the depth information set based on the same real-world scene image; The at least one depth piece of information includes: The first depth information; And / or, The first depth information, and at least one of the following: at least one depth information matching at least one frame of image preceding the first image, and at least one depth information matching at least one frame of image following the first image.

[0141] In some embodiments, the device is further configured to: The first image is obtained by performing composite rendering based on the first image information and the first depth information using a graphics processor.

[0142] In some embodiments, the device is further configured to: During the process of synthesizing and rendering based on the first image information and the first depth information, the second depth information matching the third image is simultaneously predicted and obtained. The third image is either the frame following the first image, or the third image is an image following the first image with an interval of at least one frame.

[0143] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0144] The apparatus of the above embodiments is used to implement the corresponding image processing method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0145] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image processing method described in any of the above embodiments.

[0146] Figure 8 This embodiment illustrates a more specific hardware structure diagram of an electronic device, which can be used to implement... Figure 1A The head-mounted wearable device 104 may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0147] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), image processor, neural network processor (NPU), microcontroller (MCU), programmable logic device, microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0148] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010. The memory 1020 may include volatile storage devices or non-volatile storage devices. In some embodiments, the memory 604 may include random access memory (RAM), read-only memory (ROM), optical disk, magnetic disk, hard disk, solid-state drive (SSD), flash memory, memory stick, etc.

[0149] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0150] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0151] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0152] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0153] The electronic devices described above are used to implement the corresponding image processing methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0154] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the image processing method as described in any of the above embodiments.

[0155] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0156] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the image processing method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0157] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a computer program product, which includes a computer program. In some embodiments, the computer program is executable by one or more processors to cause the processors to perform the image processing method. Corresponding to the execution entity for each step in each embodiment of the method, the processor executing the corresponding step may belong to the corresponding execution entity.

[0158] The computer program product of the above embodiments is used to cause the processor to execute the image processing method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0159] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, method, or computer program product. Therefore, this disclosure can be implemented as entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this disclosure can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0160] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (not exhaustive) of a computer-readable storage medium may include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0161] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0162] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0163] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0164] It should be understood that each block of a flowchart and / or block diagram, as well as combinations of blocks in a flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine that, when executed by a computer or other programmable data processing device, creates means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0165] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce a product comprising an instruction apparatus that implements the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0166] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable apparatus can provide a process for implementing the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0167] Furthermore, although the operations of the methods of this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be executed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0168] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0169] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0170] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0171] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0172] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0173] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for image processing, comprising: obtaining first image information, the first image information being used to generate a first image displayed by an extended reality device; obtaining first depth information matched with the first image information, the first depth information being obtained based on depth information of a real scene image matched with a predicted display time of the first image; performing synthetic rendering based on the first image information and the first depth information to obtain the first image.

2. The method of claim 1, wherein, The first image comprises at least one of a virtual object and a real object, the real object being determined based on the real scene image; and the performing synthetic rendering based on the first image information and the first depth information comprises at least one of the following: determining occlusion relationship information between the virtual object and the real object, and displaying the virtual object and the real object based on the occlusion relationship information in the first image; determining relative position relationship between the virtual object and the real object, and displaying interaction information between the virtual object and the real object based on the relative position relationship in the first image; determining light and shadow information between the virtual object and the real object, and displaying light and shadow imaging information between the virtual object and the real object based on the light and shadow information in the first image; determining spatial positioning information between the virtual object and the real object, and positioning the virtual object and the real object displayed in the first image based on the spatial positioning information.

3. The method of claim 1, further comprising: synchronously predicting the first depth information in a process of performing synthetic rendering on a second image; wherein the second image is a previous frame image of the first image, or the second image is an image before the first image and spaced by at least one frame.

4. The method of claim 3, wherein, The synchronously predicting the first depth information comprises: obtaining a predicted display time of the first image; predicting a device pose of the extended reality device to obtain a first device pose matched with the predicted display time of the first image; obtaining the real scene image based on a real scene, and obtaining the first depth information based on the real scene image and the first device pose matched with the first image.

5. The method of claim 4, wherein, The obtaining the predicted display time of the first image comprises: obtaining a predicted display time of the second image, the predicted display time of the second image being determined based on an actual display time of a previous frame image of the second image and a device frame rate of the extended reality device; obtaining the device frame rate of the extended reality device; determining the predicted display time of the first image based on the predicted display time of the second image and the device frame rate.

6. The method of claim 4, wherein, The obtaining the first depth information based on the real scene image and the first device pose matched with the first image comprises: inputting the real scene image and the first device pose into a first model, and obtaining the first depth information based on output information of the first model; The first model is configured to re-project the real scene image according to the first device pose to generate a predicted real scene image matching a predicted display time of the first image, obtain depth information of the predicted real scene image, and determine the depth information of the predicted real scene image as the first depth information.

7. The method of claim 6, wherein, the extended reality device displays a sequence of images at a device frame rate of the extended reality device, the sequence of images including the second image, the first image; the first model generates a set of depth information matching the sequence of images at the device frame rate of the extended reality device, the set of depth information including the first depth information.

8. The method of claim 7, wherein, the first model generates at least one piece of depth information in the set of depth information based on the same real scene image; the at least one piece of depth information includes: the first depth information; and / or, the first depth information, and at least one of at least one piece of depth information matching at least one frame of image before the first image, and at least one piece of depth information matching at least one frame of image after the first image.

9. The method of claim 1, further comprising: synthetic rendering, by a graphics processor, based on the first image information and the first depth information to obtain the first image.

10. The method of any one of claims 1-9, further comprising: synchronously predicting and obtaining second depth information matching a third image during the synthetic rendering based on the first image information and the first depth information; wherein the third image is a next frame of image of the first image, or the third image is an image after the first image and at least one frame apart.

11. An image processing apparatus, comprising: a first obtaining module configured to obtain first image information, the first image information being used to generate a first image displayed by an extended reality device; a second obtaining module configured to obtain first depth information matching the first image information, the first depth information being obtained based on depth information of a real scene image matching a predicted display time of the first image; a synthetic module configured to perform synthetic rendering based on the first image information and the first depth information to obtain the first image.

12. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the image processing method of any one of claims 1-10 when executing the computer program.

13. The electronic device of claim 12, wherein, The electronic device includes a wearable device.

14. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the image processing method of any one of claims 1-10.

15. A computer program product, characterised in that, Computer program instructions, when executed on a computer, cause the computer to perform the image processing method according to any one of claims 1 to 10.

Images