Image rendering methods, apparatus, equipment and media

Abstract

This application discloses an image rendering method, apparatus, device, and medium. The method includes: acquiring gaze point information of a carrier corresponding to an image in an extended reality environment; then dividing the display area of ​​the carrier based on the gaze point information to obtain multiple display sub-regions; then transforming these multiple display sub-regions to obtain multiple image sub-regions for the image; and finally rendering these multiple image sub-regions using different rendering strategies to obtain a rendered image. The technical solution of this application offers high flexibility in image rendering, improves the accuracy and efficiency of image rendering, and ensures the reliability of image rendering.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to an image rendering method, an image rendering apparatus, an electronic device, and a computer-readable medium. Background Technology

[0002] Understandably, Extended Reality (XR), as an emerging technology, is gradually entering people's field of vision and is being applied and popularized in various industries. Specifically, Extended Reality includes Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).

[0003] In extended reality image rendering schemes, related technologies typically render images of the extended reality environment directly, resulting in low flexibility in image rendering.

[0004] Therefore, improving the flexibility of image rendering is an urgent problem to be solved. Summary of the Invention

[0005] The embodiments of this application provide an image rendering method, apparatus, device, and medium, which offer high flexibility in image rendering, improve the accuracy and efficiency of image rendering, and ensure the reliability of image rendering.

[0006] In a first aspect, embodiments of this application provide an image rendering method, the method comprising: acquiring gaze point information of a carrier corresponding to an image in an extended reality environment; dividing the display area of ​​the carrier based on the gaze point information to obtain multiple display sub-regions; converting the multiple display sub-regions to obtain multiple image sub-regions for the image; and rendering the multiple image sub-regions using different rendering strategies to obtain a rendered image.

[0007] Secondly, embodiments of this application provide an image rendering apparatus, the apparatus comprising: an acquisition module configured to acquire gaze point information of a carrier corresponding to an image in an extended reality environment; a segmentation module configured to segment the display area of ​​the carrier based on the gaze point information to obtain multiple display sub-regions; a conversion module configured to convert based on the multiple display sub-regions to obtain multiple image sub-regions for the image; and a rendering module configured to render the multiple image sub-regions using different rendering strategies to obtain a rendered image.

[0008] In one embodiment of this application, based on the aforementioned scheme, the gaze point information includes the position information of the gaze point in the display area of ​​the carrier; the division module is specifically configured to: obtain the grid layout information of the grid contained in the display area; and divide the display area based on the grid layout information and the position information of the gaze point to obtain multiple display sub-regions.

[0009] In one embodiment of this application, based on the foregoing scheme, the plurality of display sub-regions include a first display sub-region and a second display sub-region; the division module is further specifically configured to: select a first range area around the position represented by the position information of the gaze point from the display area based on the grid layout information; divide the first range area into the first display sub-region, and divide the other areas in the display area other than the first display sub-region into the second display sub-region.

[0010] In one embodiment of this application, based on the foregoing scheme, the conversion module is specifically configured to: obtain the position information of each display sub-region in the display area; convert the position information of each display sub-region based on the size information of the image to obtain the position conversion information of each display sub-region; and determine the image sub-region corresponding to each display sub-region from the image based on the position conversion information of each display sub-region.

[0011] In one embodiment of this application, based on the foregoing scheme, the position information of each display sub-region includes the position information of multiple display points located on the boundary of each display sub-region; the conversion module is further specifically configured to: obtain the position information of multiple display points corresponding to the same display sub-region; and convert the position information of each display point based on the size information of the image to obtain the position conversion information of the same display sub-region.

[0012] In one embodiment of this application, based on the foregoing scheme, the conversion module is further configured as follows: if the plurality of display sub-regions include a first display sub-region and a second display sub-region, and the second display sub-region is another region in the display region besides the first display sub-region, then the position information of the first display sub-region is converted based on the size information of the image to obtain position conversion information of the first display sub-region, and the position conversion information of the second display sub-region is obtained based on the position conversion information of the first display sub-region.

[0013] In one embodiment of this application, based on the aforementioned scheme, the rendering module is specifically configured to: render the plurality of image sub-regions at different resolutions to generate a rendering region image corresponding to each image sub-region; and fuse the plurality of rendering region images to obtain a rendering image.

[0014] In one embodiment of this application, based on the foregoing scheme, the plurality of image sub-regions include a first image sub-region and a second image sub-region; the rendering module is further specifically configured to: input the image into a first rendering model, determine the first image sub-region from the image through the first rendering model, render the first image sub-region using a first resolution, and generate a first rendering region image corresponding to the first image sub-region; and input the image into a second rendering model, determine the second image sub-region from the image through the second rendering model, render the second image sub-region using a second resolution, and generate a second rendering region image corresponding to the second image sub-region.

[0015] In one embodiment of this application, based on the foregoing scheme, the rendering module is further configured to: crop out a first region image corresponding to the first image sub-region from the image using the first rendering model, and render based on the first resolution and the pixel values ​​contained in the first region image to generate a first rendering region image.

[0016] In one embodiment of this application, based on the foregoing scheme, the rendering module is further configured to: determine the pixel values ​​contained in the first image sub-region from the image using the second rendering model, and set the determined pixel values ​​to zero to obtain a second region image corresponding to the second image sub-region; and render based on the second resolution and the pixel values ​​contained in the second region image to generate a second rendering region image.

[0017] In one embodiment of this application, based on the foregoing scheme, the rendering module is specifically configured as follows: obtaining an image sub-region group based on the plurality of image sub-regions, the image sub-region group including two adjacent image sub-regions; rendering the two image sub-regions in the image sub-region group using different rendering strategies to generate a rendering region image corresponding to each image sub-region in the image sub-region group; obtaining adjacent image regions of the two image sub-regions in the image sub-region group and generating rendering region images corresponding to the adjacent image regions; fusing the rendering region images corresponding to the adjacent image regions and the rendering region images corresponding to each image sub-region in the image sub-region group to obtain a rendered image.

[0018] In one embodiment of this application, based on the foregoing scheme, the rendering module is further configured to: obtain rendering strategies for rendering two image sub-regions in the image sub-region group respectively; generate a rendering parameter value range based on the rendering parameter values ​​corresponding to the two rendering strategies respectively, and generate a target rendering parameter value located in the rendering parameter value range; and use the target rendering parameter value to render the adjacent image region to generate a rendering region image corresponding to the adjacent image region.

[0019] In one embodiment of this application, based on the foregoing scheme, the rendering module is further configured to: extract candidate region images corresponding to the adjacent image regions from the rendering region image corresponding to each image sub-region in the image sub-region group; and generate the rendering region image corresponding to the adjacent image regions based on the two candidate region images and the weight value corresponding to each candidate region image.

[0020] Thirdly, embodiments of this application provide an electronic device, including one or more processors; and a memory for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the image rendering method described above.

[0021] Fourthly, embodiments of this application provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the image rendering method described above.

[0022] Fifthly, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the image rendering method described above.

[0023] In the technical solution provided by the embodiments of this application: the display area of ​​the carrier corresponding to the image in the extended reality environment is divided by utilizing the gaze point information of the carrier being gazed at, and based on the corresponding transformation of the divided display sub-regions, multiple image sub-regions for the image are obtained, and then multiple image sub-regions are rendered using different rendering strategies. This achieves regional rendering of images in the extended reality environment, which is highly flexible; and because of the different rendering strategies, the image quality of some regions is relatively high, while the image quality of some regions is relatively low, which improves the rendering accuracy to a certain extent. At the same time, the computational resources consumed by the relatively high image quality of some regions are relatively high, and the computational resources consumed by the relatively low image quality of some regions are relatively low, thus improving the rendering efficiency to a certain extent and ensuring the reliability of image rendering.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating an exemplary implementation environment in which the technical solutions of the embodiments of this application can be applied.

[0026] Figure 2 This is a flowchart illustrating an exemplary embodiment of the image rendering method of this application.

[0027] Figure 3 This is a schematic diagram of a display sub-region shown in an exemplary embodiment of this application.

[0028] Figure 4 This is a flowchart illustrating an image rendering method as shown in another exemplary embodiment of this application.

[0029] Figure 5A This is a schematic diagram illustrating adjacent image regions in an exemplary embodiment of this application.

[0030] Figure 5B This is a schematic diagram illustrating adjacent image regions, as shown in another exemplary embodiment of this application.

[0031] Figure 6 This is a flowchart illustrating an image rendering method as shown in another exemplary embodiment of this application.

[0032] Figure 7 This is a schematic diagram illustrating an image rendering method according to an exemplary embodiment of this application.

[0033] Figure 8 This is a block diagram illustrating an image rendering apparatus according to an exemplary embodiment of this application.

[0034] Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application embodiments. Detailed Implementation

[0035] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments identical to those described in this application. Rather, they are merely examples of apparatuses and methods identical to some aspects of this application as detailed in the appended claims.

[0036] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0037] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0038] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0039] It should be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0040] Understandably, extended reality, as an emerging technology, is gradually entering people's field of vision and is being applied and popularized in various industries. Extended reality technology specifically includes virtual reality, augmented reality, and mixed reality technologies. In extended reality image rendering schemes, these technologies typically render images directly from the extended reality environment, resulting in relatively low flexibility in image rendering.

[0041] Therefore, to improve the flexibility of image rendering, this application provides an image rendering scheme. Please refer to... Figure 1 , Figure 1 This is a schematic diagram of an implementation environment related to this application. The implementation environment mainly includes a terminal device 101 and a server 102; wherein:

[0042] Terminal devices 101 include, but are not limited to, extended reality devices (virtual reality devices, augmented reality devices, mixed reality devices, etc.), mobile phones, computers (tablets, laptops, desktop computers, etc.), smart home devices (televisions, refrigerators, air conditioners, washing machines, robot vacuums, etc.), smart wearable devices (bracelets, watches, etc.).

[0043] Server 102 can be a standalone physical server, or a server cluster or distributed system consisting of multiple physical servers. The server cluster or distributed system includes cloud servers used to provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0044] It is understood that terminal device 101 and server 102 establish a communication connection via a wired or wireless network. Exemplarily, the wireless or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any other network, including but not limited to a Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), mobile, wired or wireless networks, private networks, or any combination of virtual private networks.

[0045] In one embodiment of this application, image rendering can be performed by server 102. Specifically, server 102 obtains the gaze point information of the carrier corresponding to the image in the extended reality environment, then divides the display area of ​​the carrier based on the gaze point information to obtain multiple display sub-regions, then transforms the multiple display sub-regions to obtain multiple image sub-regions for the image, and then renders the multiple image sub-regions using different rendering strategies to obtain the rendered image.

[0046] In other embodiments of this application, the image rendering method may be executed by the terminal device 101 alone or by the terminal device 101 and the server 102 together. In practical applications, the executing entity of the image rendering method can be flexibly adjusted according to the specific application scenario.

[0047] It should be made clear that, Figure 1 The number of terminal devices 101 and servers 102 shown is merely illustrative; any number of terminal devices 101 and servers 102 can be used as needed.

[0048] Understandable Figure 1The terminal device 101 and server 102 support an extended reality system. Specifically, objects (such as users) can be in an extended reality environment through the terminal device 101, while the server 102 provides corresponding background services for the terminal device 101.

[0049] Extended reality systems can be applied to various scenarios, such as:

[0050] In entertainment scenarios, players can construct entertainment environments (such as game environments) using extended reality devices (such as head-mounted displays (HMDs)) to immerse themselves in the entertainment environment. By implementing the solution in this application, different rendering methods are applied to the images in the entertainment environment, ensuring the reliability of image rendering and enhancing the player's entertainment experience.

[0051] In the learning scenario, students can simulate the learning environment using augmented reality devices (such as glasses) to immerse themselves in the learning environment (such as a flight environment, a fire rescue environment, etc.). By implementing the solution in this application, different rendering methods are applied to the images in the learning environment, ensuring the reliability of image rendering and improving the learning experience for students.

[0052] It should be noted that in the specific implementation of this application, user-related data such as gaze point information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0053] The following details the various implementation details of the technical solutions in the embodiments of this application:

[0054] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the image rendering method of this application. Figure 2 As shown, this image rendering method includes at least S201 to S204, which are described in detail below:

[0055] S201, Obtain gaze point information of the carrier corresponding to the image in the extended reality environment.

[0056] In the embodiments of this application, the image in the extended reality environment refers to the image generated based on extended reality, which can be one or more frames of an image in a video generated based on extended reality.

[0057] In the embodiments of this application, the carrier refers to the hardware on which the image is displayed, including but not limited to at least one of a display screen, a screen, and an optical medium. The display screen is usually the most common carrier. The screen is usually used to realize the display of projected images based on extended reality. The optical medium is usually used to realize the display of holographic images based on extended display, and it can be a glass plate, optical element, etc.

[0058] In this embodiment, the gaze point information refers to information related to the gaze point on the carrier, including but not limited to the gaze point's location information (i.e., the position of the gaze point in the display area of ​​the carrier, which can be represented by (X, Y)) and the gaze point's duration information (i.e., the duration of the gaze point's stay on the carrier, which is for gaze points at the same location, and can be represented by (X, Y)). (represented by) at least one of the following.

[0059] In one embodiment of this application, the process of obtaining gaze point information of the carrier corresponding to the image in the extended reality environment in S201 may include:

[0060] The interpupillary distance information of the subject is collected using eye-tracking technology;

[0061] Based on interpupillary distance information, obtain the gaze point information of the carrier corresponding to the image in the extended reality environment.

[0062] That is, in an optional embodiment, eye detection technology is used to collect the interpupillary distance information of an object (e.g., a user). The eye detection technology includes, but is not limited to, at least one of line-of-sight detection and human eye detection. Then, the interpupillary distance information is used to obtain the gaze point information of the carrier corresponding to the image in the extended reality environment.

[0063] In one of the optional embodiments, the pupillary distance information of the object can be collected by eye detection technology: by collecting the pupillary distance image of the object through an eye detection device (such as an infrared device, an image acquisition device, etc.) deployed in the extended reality system, and then extracting the pupillary distance information from the pupillary distance image.

[0064] In one optional embodiment, obtaining the gaze point information of the carrier corresponding to the image in the extended reality environment based on the interpupillary distance (IPD) information can be achieved by: acquiring standard IPD information corresponding to the collected IPD information; comparing the collected IPD information with the acquired standard IPD information; and then obtaining the gaze point information of the carrier corresponding to the image in the extended reality environment based on the comparison result. The standard IPD information can be obtained through pre-setting, typically with one standard IPD information set for each type of standard face.

[0065] By implementing this optional embodiment, eye detection technology can be used to easily and accurately obtain gaze point information of the carrier corresponding to the image in the extended reality environment, thereby providing strong support for the division of the display area.

[0066] S202, the display area of ​​the carrier is divided based on the gaze point information to obtain multiple display sub-regions.

[0067] In this embodiment, the gaze point information of the carrier corresponding to the image in the extended reality environment is obtained. Then, the gaze point information can be used to divide the display area of ​​the carrier, thereby obtaining multiple display sub-regions for the carrier.

[0068] It is understood that the carrier has a display area for displaying images in an extended display environment; therefore, in the embodiments of this application, the display area of ​​the carrier is divided to obtain multiple display sub-regions for the carrier, wherein multiple display sub-regions refer to at least two display sub-regions, and each display sub-region is a part of the entire display area of ​​the carrier.

[0069] As described in the foregoing embodiments, the fixation point information includes the location information of the fixation point, the duration information of the fixation point, etc.; in one embodiment of this application, the fixation point information is described as including location information. Accordingly, the process of dividing the display area of ​​the carrier based on the fixation point information to obtain multiple display sub-regions in S202 may include:

[0070] Obtain the grid layout information of the grids contained in the display area;

[0071] The display area is divided based on grid layout information and gaze point location information to obtain multiple display sub-regions.

[0072] That is, in an optional embodiment, the grid layout information of the grid contained in the display area is first obtained, and then the display area is divided using the grid layout information and the position information of the gaze point, thereby obtaining multiple display sub-regions.

[0073] In one of the optional embodiments, the display area includes multiple grids. In other words, multiple grids together form the display area. Correspondingly, the grid layout information refers to the arrangement information of the multiple grids that make up the display area, which can usually be represented by N rows and M columns.

[0074] By implementing this optional embodiment, the division of the display area can be achieved simply and accurately using the position information of the gaze point and the grid layout information of the grid contained in the display area.

[0075] As described in the foregoing embodiments, the multiple display sub-regions include at least two display sub-regions; in one embodiment of this application, the multiple display sub-regions are referred to as two display sub-regions, namely the first display sub-region and the second display sub-region. Accordingly, the process of dividing the display area based on grid layout information and gaze point position information to obtain multiple display sub-regions may include:

[0076] Based on grid layout information, a first range area surrounding the position represented by the location information of the gaze point is selected from the display area;

[0077] The first range area is divided into a first display sub-area, and the other areas in the display area other than the first display sub-area are divided into a second display sub-area.

[0078] In other words, in an optional embodiment, a first range area around / near the location represented by the position information of the gaze point is first selected from the display area using grid layout information. Then, the first range area is divided into a first display sub-region, and the remaining area in the display area other than the first display sub-region is divided into a second display sub-region. Thus, the first display sub-region and the second display sub-region are obtained. In other words, a first display sub-region of a corresponding size is determined from the display area with the gaze point as the center, and the remaining part of the display area is the second display sub-region.

[0079] In one of the optional embodiments, the size of the first range area can be one-half, one-third, one-quarter, etc. of the display area size, and the shape of the first range area can be regular (e.g., polygon, circle, etc.) or irregular. In practical applications, the size and shape of the first range area can be flexibly adjusted according to the specific application scenario.

[0080] By implementing this optional embodiment, the display area is divided into two sub-display areas. The division process is simple and easy to implement, improving the efficiency of display area division and thus improving rendering efficiency.

[0081] S203, based on multiple display sub-regions, performs conversion to obtain multiple image sub-regions for the image.

[0082] In this embodiment, multiple display sub-regions for the carrier are obtained. These multiple display sub-regions can then be transformed to obtain multiple image sub-regions for the image. That is, through the corresponding transformation of the multiple display sub-regions, an image sub-region corresponding to each display sub-region can be obtained. There is a one-to-one relationship between the display sub-regions and the image sub-regions, and each image sub-region is a part of the entire image area.

[0083] In one embodiment of this application, the process of converting multiple display sub-regions to obtain multiple image sub-regions for an image in S203 may include:

[0084] Obtain the position information of each display sub-region within the display area;

[0085] Based on the image size information, the position information of each display sub-region is transformed to obtain the position transformation information of each display sub-region;

[0086] Based on the position transformation information of each display sub-region, the image sub-region corresponding to each display sub-region is determined from the image.

[0087] That is, in the optional embodiment, the position information of each display sub-region in the display area is first obtained, and then the position information of each display sub-region is transformed using the size information of the image to obtain the position transformation information of each display sub-region. In other words, the position of each display sub-region in the display area is transformed to its position in the image. Then, the position transformation information of each display sub-region is used to determine the image sub-region corresponding to each display sub-region from the image, thereby obtaining multiple image sub-regions for the image.

[0088] In the optional embodiment, the image size information refers to information related to the image size, which can usually be represented by W (width) and H (height).

[0089] In one of the optional embodiments, since the image is in an extended reality environment, the position transformation information of each display sub-region is also based on the extended reality environment. That is, there is a coordinate system in the extended reality environment, and the transformation of the position information of each display sub-region is essentially a transformation to the image based on the coordinate system of the extended reality environment.

[0090] For example, consider multiple display sub-regions, including a first display sub-region and a second display sub-region. Let the position information of the first display sub-region in the display area be Di1_pi, and the position information of the second display sub-region in the display area be Di2_pi. Then, by converting the position information Di1_pi, we can obtain the position conversion information Di1_pi' of the first display sub-region. Similarly, by converting the position information Di2_pi, we can obtain the position conversion information Di2_pi' of the second display sub-region. The position conversion information Di1_pi' of the first display sub-region is the position information of the first image sub-region corresponding to the first display sub-region in the image. Therefore, based on the position conversion information Di1_pi', the first image sub-region corresponding to the first display sub-region can be determined from the image. Similarly, the position conversion information Di2_pi' of the second display sub-region is the position information of the second image sub-region corresponding to the second display sub-region in the image. Therefore, based on the position conversion information Di2_pi', the second image sub-region corresponding to the second display sub-region can be determined from the image.

[0091] By implementing this optional embodiment, the position information of each display sub-region can be transformed using the image size information, thus enabling the simple and accurate determination of the image sub-region corresponding to each display sub-region, providing strong support for different rendering methods.

[0092] In one embodiment of this application, the location information of each display sub-region includes the location information of multiple display points located on the boundary of each display sub-region.

[0093] It is understood that each display sub-region includes a first number of display points, and the boundary of each display sub-region includes a second number of display points, wherein the second number is less than the first number. The first number corresponding to different display sub-regions may be the same or different, and the second number corresponding to the boundary of different display sub-regions may be the same or different.

[0094] It should be clear that the boundary of the displayed sub-region can be the outline of the displayed sub-region. For example, please refer to... Figure 3 The multiple display sub-regions include a first display sub-region 301 and a second display sub-region 302. The first display sub-region 301 (the area shown by the diagonal line) is rectangular, and the second display sub-region 302 (the area shown by the horizontal line) is annular. The boundary of the first display sub-region 301 is the outer contour line, which is also the boundary of the second display sub-region 302, but it corresponds to the inner contour line of the second display sub-region 302. In addition, the boundary of the second display sub-region 302 may also include the outer contour line.

[0095] In one optional embodiment, obtaining the position information of each display sub-region within the display area can be achieved by: obtaining a third number of display points from the boundary of the same display sub-region, determining the position information of each display point obtained for the boundary of the same display sub-region, and combining the position information corresponding to multiple display points for the boundary of the same display sub-region to obtain the position information corresponding to each display sub-region. Thus, the position information of each display sub-region includes the position information of multiple display points located on the boundary of each display sub-region. It is understood that the third number corresponding to the boundaries of different display sub-regions can be the same or different.

[0096] Accordingly, the process of transforming the position information of each display sub-region based on the image size information to obtain the position transformation information of each display sub-region may include:

[0097] Obtain the position information of multiple display points corresponding to the same display sub-region;

[0098] The position information of each display point is transformed based on the size information of the image to obtain the position transformation information of the same display sub-region.

[0099] That is, in the optional embodiment, when the position information of each display sub-region includes the position information of multiple display points located on the boundary of each display sub-region, the position information of multiple display points corresponding to the same display sub-region can be obtained from the position information of each display sub-region. Then, the position information of each display point corresponding to the same display sub-region is converted using the size information of the image, thereby obtaining the position conversion information of each display sub-region.

[0100] By implementing this optional embodiment, the position information of a certain number of display points corresponding to the same display sub-region can be converted, thus easily and accurately obtaining the position conversion information of the same display sub-region.

[0101] In one embodiment of this application, the process of converting the position information of each display sub-region based on the image size information to obtain the position conversion information of each display sub-region may include:

[0102] If multiple display sub-regions include a first display sub-region and a second display sub-region, and the second display sub-region is another region in the display area besides the first display sub-region, then the position information of the first display sub-region is transformed based on the size information of the image to obtain the position transformation information of the first display sub-region, and the position transformation information of the second display sub-region is obtained based on the position transformation information of the first display sub-region.

[0103] That is, in the optional embodiment, if the multiple display sub-regions in the aforementioned embodiment include a first display sub-region and a second display sub-region, the position information of the first display sub-region can be converted using the size information of the image to obtain the position conversion information of the first display sub-region. Then, since the second display sub-region is the other area in the display area besides the first display sub-region, in other words, the first display sub-region and the second display sub-region are related, the position conversion information of the second display sub-region can be calculated using the position conversion information of the first display sub-region.

[0104] In other embodiments, as described in the foregoing embodiments, the position information of the second display sub-region can be converted using the size information of the image to obtain the position conversion information of the second display sub-region; specifically, the position information of multiple display points corresponding to the second display sub-region is obtained, and then the position information of each display point is converted using the size information of the image to obtain the position conversion information of the second display sub-region.

[0105] By implementing this optional embodiment, when the display area is divided into two sub-display areas, the corresponding position conversion information can be calculated by utilizing the relationship between the two sub-display areas, thus providing high flexibility.

[0106] S204: Render multiple image sub-regions using different rendering strategies to obtain the rendered image.

[0107] In this embodiment, multiple image sub-regions are obtained for an image. Then, different rendering strategies can be applied to these multiple image sub-regions to obtain a rendered image.

[0108] The rendering strategies in this application include, but are not limited to, at least one of resolution, color, and brightness, where resolution, color, and brightness are three rendering parameters. Different rendering strategies in this application refer to different rendering parameter values ​​corresponding to the same rendering parameter.

[0109] As described in the foregoing embodiments, the rendering strategy includes resolution, color, brightness, etc.; in one embodiment of this application, the rendering strategy is described using resolution as an example. Accordingly, the process of rendering multiple image sub-regions using different rendering strategies in S204 to obtain a rendered image may include:

[0110] Render multiple image sub-regions at different resolutions to generate a rendering area image corresponding to each image sub-region;

[0111] Multiple rendering region images are merged to obtain a rendered image.

[0112] That is, in an optional embodiment, multiple image sub-regions are rendered at different resolutions to generate a rendering area image corresponding to each image sub-region, and then the multiple rendering area images are fused to obtain a rendered image.

[0113] For example, consider two image sub-regions, referred to as the first image sub-region and the second image sub-region, where the first image sub-region corresponds to a first resolution n1, and the second image sub-region corresponds to a second resolution n2, where n1 ≠ n2. Then, the first image sub-region can be rendered using the first resolution n1 to generate a rendered image corresponding to the first image sub-region, and the second image sub-region can be rendered using the second resolution n2 to generate a rendered image corresponding to the second image sub-region. Finally, the rendered images of the first and second image sub-regions are merged to obtain the final rendered image.

[0114] In one optional embodiment, rendering multiple image sub-regions at different resolutions to generate a rendered region image corresponding to each image sub-region can be achieved by using a rendering model to render multiple image sub-regions at different resolutions, thereby generating a rendered region image corresponding to each image sub-region. This rendering model-based approach offers high accuracy and a high degree of intelligence.

[0115] In one optional embodiment, fusing multiple rendering region images to obtain a rendered image can be achieved by fusing multiple rendering region images using a fusion model. This fusion method, using a fusion model, offers high accuracy and a high degree of intelligence.

[0116] It is understandable that the aforementioned rendering model and fusion model can be integrated into the same model or they can be two separate models. In practical applications, they can be flexibly adjusted according to the specific application scenario.

[0117] By implementing this optional embodiment, the fusion of multiple rendering region images can be easily and accurately generated into a rendered image.

[0118] In one embodiment of this application, the aforementioned plurality of image sub-regions are referred to as two image sub-regions, respectively called the first image sub-region and the second image sub-region. Accordingly, the process of rendering the plurality of image sub-regions at different resolutions to generate a rendered region image corresponding to each image sub-region may include:

[0119] An image is input into a first rendering model, and a first image sub-region is determined from the image using the first rendering model. The first image sub-region is then rendered using a first resolution to generate a first rendering region image corresponding to the first image sub-region.

[0120] The image is input into the second rendering model, and the second image sub-region is determined from the image through the second rendering model. The second image sub-region is rendered using the second resolution to generate the second rendering region image corresponding to the second image sub-region.

[0121] That is, in the optional embodiment, multiple rendering models are used to generate rendering area images corresponding to multiple image sub-regions in parallel; specifically, a first image sub-region is determined from the input image using a first rendering model, and the first image sub-region is rendered using a first resolution to generate a first rendering area image corresponding to the first image sub-region; and a second image sub-region is determined from the input image using a second rendering model, and the second image sub-region is rendered using a second resolution to generate a second rendering area image corresponding to the second image sub-region.

[0122] By implementing this optional embodiment, the parallel generation of the first rendering region image and the second rendering region image is achieved with the help of the first rendering model and the second rendering model, resulting in high rendering efficiency.

[0123] In one embodiment of this application, the process of determining a first image sub-region from an image using a first rendering model, rendering the first image sub-region using a first resolution, and generating a first rendered region image corresponding to the first image sub-region may include:

[0124] The first region image is generated by cropping a first image sub-region from the image using the first rendering model, and by rendering based on the first resolution and the pixel values ​​contained in the first region image.

[0125] That is, in the optional embodiment, the first rendering model first crops out the first region image corresponding to the first image sub-region from the input image, and then renders it based on the first resolution and the pixel values ​​contained in the first region image to generate the first rendering region image and output the first rendering region image.

[0126] In one embodiment of this application, the process of determining a second image sub-region from an image using a second rendering model, rendering the second image sub-region using a second resolution, and generating a second rendering region image corresponding to the second image sub-region may include:

[0127] The second rendering model determines the pixel values ​​contained in the first image sub-region from the image, and sets the determined pixel values ​​to zero to obtain the second region image corresponding to the second image sub-region. The second rendering region image is generated by rendering based on the second resolution and the pixel values ​​contained in the second region image.

[0128] That is, in the optional embodiment, the second rendering model first determines the pixel values ​​contained in the first image sub-region from the input image, and sets the determined pixel values ​​to zero, thereby obtaining the second region image corresponding to the second image sub-region. Then, it renders based on the second resolution and the pixel values ​​contained in the second region image to generate the second rendering region image and outputs the second rendering region image.

[0129] Accordingly, in one embodiment of this application, the process of fusing multiple rendering region images to obtain a rendered image may include:

[0130] The first and second rendering region images are input into the fusion model, and the fusion model is used to fuse the first and second rendering region images to generate a rendered image.

[0131] That is, in the optional embodiment, the fusion model fuses the input first rendering region image and the second rendering region image to obtain a rendered image, and outputs the rendered image for corresponding display for the object to view.

[0132] By implementing this optional embodiment, the first rendering region image and the second rendering region image are fused using a fusion model, resulting in high fusion efficiency.

[0133] This application embodiment realizes regional rendering of images in extended reality environment, which is highly flexible; and due to different rendering strategies, the image quality of some regions is relatively high and the image quality of some regions is relatively low, which improves the rendering accuracy to a certain extent. At the same time, the computational resources consumed by the relatively high image quality of some regions are relatively high, and the computational resources consumed by the relatively low image quality of some regions are relatively low, thus improving the rendering efficiency to a certain extent and ensuring the reliability of image rendering.

[0134] In one embodiment of this application, see Figure 4 , Figure 4 Is Figure 2 The flowchart shown is based on the proposed image rendering method. S204 may include S401 to S404. Figure 4 For detailed information on S201 to S203 shown, please refer to [link / reference]. Figure 2 S201 to S203 shown will not be described in detail here. The detailed description of S401 to S404 is as follows:

[0135] S401, an image sub-region group is obtained based on multiple image sub-regions, and the image sub-region group includes two adjacent image sub-regions.

[0136] In this embodiment of the application, two adjacent image sub-regions can be combined into an image sub-region group, that is, an image sub-region group includes two adjacent image sub-regions; if there are only two image sub-regions, the number of image sub-region groups is usually one; if there are three or more image sub-regions, the number of image sub-region groups is usually at least two.

[0137] S402, render two image sub-regions in the image sub-region group using different rendering strategies to generate the rendering region image corresponding to each image sub-region in the image sub-region group.

[0138] In this embodiment of the application, an image sub-region group is obtained. Then, two image sub-regions in the image sub-region group can be rendered using different rendering strategies to generate a rendering area image corresponding to each image sub-region in the image sub-region group. The rendering strategies corresponding to different image sub-region groups can be the same or different.

[0139] S403, obtain the adjacent image regions of two image sub-regions in the image sub-region group, and generate the rendering region image corresponding to the adjacent image regions.

[0140] In this embodiment, a rendering region image corresponding to each image sub-region in the image sub-region group is generated. Then, the adjacent image regions of two image sub-regions in the image sub-region group can be obtained, and the rendering region image corresponding to the adjacent image regions can be generated.

[0141] Optionally, adjacent image regions can be two adjacent image sub-regions within a group of image sub-regions; for example, see [link to relevant documentation]. Figure 5A There is a blank area 503 between the first image sub-region 501 (the area shown by the diagonal line) and the second image sub-region 502 (the area shown by the horizontal line), where area 503 is the adjacent image area of ​​the first image sub-region 501 and the second image sub-region 502.

[0142] Alternatively, adjacent image regions can be obtained by cropping portions of two image sub-regions from a group of image sub-regions; for example, please refer to [link to relevant documentation]. Figure 5B There is no empty image area between the first image sub-region 501 (the area shown by the diagonal line) and the second image sub-region 502 (the area shown by the horizontal line), that is, the inner contour line of the first image sub-region 501 overlaps with the outer contour line of the second image sub-region 502. At this time, an area close to the outer contour of the first image sub-region 501 and an area close to the inner contour of the second image sub-region 502 can be extracted. The combined extracted area 5012 (i.e., the area shown by both the horizontal and diagonal lines) is the adjacent image area of ​​the first image sub-region 501 and the second image sub-region 502.

[0143] In one embodiment of this application, the process of generating rendering region images corresponding to adjacent image regions in S403 may include:

[0144] Obtain the rendering strategy used to render two image sub-regions in the image sub-region group respectively;

[0145] Based on the rendering parameter values ​​corresponding to the two rendering strategies, a range of rendering parameter values ​​is generated, and a target rendering parameter value located within the range of rendering parameter values ​​is generated.

[0146] The target rendering parameter values ​​are used to render adjacent image regions, generating rendering region images corresponding to the adjacent image regions.

[0147] That is, in an optional embodiment, a rendering strategy for rendering two image sub-regions in the image sub-region group is first obtained, then a rendering parameter value range is generated using the rendering parameter values ​​corresponding to the two rendering strategies, and a target rendering parameter value located in the rendering parameter value range is generated. Then, the target rendering parameter value is used to render the adjacent image regions, thereby generating the rendering region image corresponding to the adjacent image regions.

[0148] For example, consider a group of image sub-regions, including a first image sub-region and a second image sub-region, with a rendering strategy based on resolution. Let the first resolution corresponding to the first image sub-region be n1, and the second resolution corresponding to the second image sub-region be n2, where n1 > n2. Then, a resolution interval [n2, n1] can be generated. Correspondingly, a target resolution n3 (i.e., n1 > n3 > n2) located within the resolution interval [n2, n1] can be generated. Then, the target resolution n3 is used to render adjacent image regions to generate the rendered region images corresponding to the adjacent image regions.

[0149] By implementing this optional embodiment, the target rendering parameter value is generated using the rendering parameter value corresponding to the rendering strategy, and the target rendering parameter value is used to render adjacent image regions, thus enabling the simple and accurate generation of rendering region images corresponding to adjacent image regions.

[0150] In one embodiment of this application, the process of generating rendering region images corresponding to adjacent image regions in S403 may include:

[0151] Extract candidate region images corresponding to adjacent image regions from the rendered region image corresponding to each image sub-region group;

[0152] The rendering region image corresponding to the adjacent image region is generated by calculating the weight value of each candidate region image based on two candidate region images.

[0153] That is, in an optional embodiment, the rendering region image corresponding to each image sub-region in the image sub-region group is extracted, and the candidate region image corresponding to the adjacent image region is extracted. Then, the two candidate region images and the weight value corresponding to each candidate region image are used to calculate and generate the rendering region image corresponding to the adjacent image region.

[0154] For example, consider a group of image sub-regions, including a first image sub-region and a second image sub-region, with a rendering strategy based on resolution. Assume the first image sub-region is rendered at a first resolution n1, resulting in a rendered region image corresponding to the first image sub-region. Similarly, the second image sub-region is rendered at a second resolution n2, resulting in a rendered region image corresponding to the second image sub-region. Then, candidate region images ca1 corresponding to adjacent image regions can be extracted from the rendered region image corresponding to the first image sub-region, and candidate region images ca2 corresponding to adjacent image regions can be extracted from the rendered region image corresponding to the second image sub-region. Next, a weighted summation operation (ca1×a1+ca2×a2) is performed using candidate region images ca1, their corresponding weight values ​​a1, ca2, and a2, to generate the rendered region images corresponding to adjacent image regions. Optionally, a1+a2=1.

[0155] In other embodiments, candidate region images corresponding to adjacent image regions can be extracted from the rendered region image corresponding to a certain image sub-region in the image sub-region group.

[0156] By implementing this optional embodiment, there is no need for separate rendering. Instead, the rendered area image can be cropped from the rendered area image, and the cropped area can be processed accordingly. This allows for the simple and accurate generation of rendered area images corresponding to adjacent image areas.

[0157] S404, fuse the rendering region images corresponding to adjacent image regions and the rendering region images corresponding to each image sub-region in the image sub-region group to obtain the rendered image.

[0158] In this embodiment, the rendering region image corresponding to the adjacent image region and the rendering region image corresponding to each image sub-region in the image sub-region group are generated. Then, the rendering region images corresponding to the adjacent image regions and the rendering region images corresponding to each image sub-region in the image sub-region group are fused to obtain the rendering image.

[0159] It is understandable that in the aforementioned implementation, when a candidate region image ca1 corresponding to an adjacent image region is extracted from the rendering region image corresponding to the first image sub-region, the candidate region image ca1 needs to be discarded from the rendering region image corresponding to the first image sub-region during fusion. Similarly, when a candidate region image ca2 corresponding to an adjacent image region is extracted from the rendering region image corresponding to the second image sub-region, the candidate region image ca2 needs to be discarded from the rendering region image corresponding to the second image sub-region during fusion.

[0160] It is understandable that in the aforementioned implementation, when a candidate region image corresponding to an adjacent image region is extracted from the rendering region image corresponding to a certain image sub-region in the image sub-region group, the candidate region image needs to be discarded from the rendering region image corresponding to that certain image sub-region during fusion.

[0161] In practical applications, whether to discard the corresponding rendered area image during fusion can be flexibly adjusted according to the specific application scenario. If it is necessary to discard, then fusion can be performed simply. If it is not necessary to discard, then overlapping fusion can be performed.

[0162] In the rendering process of multiple image sub-regions using different rendering strategies, the adjacent image regions between different image sub-regions are taken into consideration. Then, the rendering region images corresponding to the adjacent image regions and the rendering region images corresponding to each image sub-region are fused together to obtain the rendered image, making the transition between adjacent areas more natural and smooth, further improving the rendering accuracy, rendering effect, and user experience.

[0163] The following provides a detailed description of specific scenarios in the embodiments of this application:

[0164] Please see Figure 6 , Figure 6 This is a flowchart illustrating an embodiment of the image rendering method of this application. Figure 6 As shown, this image rendering method includes at least S601 to S609, which are described in detail below:

[0165] S601, Obtain the position information of the gaze point of the carrier corresponding to the image in the extended reality environment.

[0166] For example, let the fixation point be... Its position in the corresponding display area of ​​the carrier (e.g., display screen) is (X, Y).

[0167] S602, Obtain the grid layout information of the grids contained in the display area.

[0168] Following the previous example, let's assume that the obtained grid layout information of the grid contained in the display area is N×M, that is, N rows and M columns.

[0169] S603, based on grid layout information, selects a first range area around the position represented by the position information of the gaze point from the display area.

[0170] S604, the first range area is divided into a first display sub-area, and the other areas in the display area other than the first display sub-area are divided into a second display sub-area.

[0171] Following the previous example, let the first range area be rectangular, that is, the first display sub-area is rectangular, and the size of the first display sub-area is (N / 2)×(M / 2), that is, N / 2 rows and M / 2 columns. Then the outer contour of the first display sub-area must satisfy: upper boundary (Y–N / 2)≥0, lower boundary (Y+N / 2)≤N, left boundary (X–M / 2)≥0, and right boundary (X+M / 2)≤M; correspondingly, the second display sub-area is the other area in the display area besides the second display sub-area.

[0172] At this point, the display area is divided into a first display sub-area and a second display sub-area.

[0173] S605, obtain the position information of the first display sub-region in the display area, and obtain the position information of the second display sub-region in the display area.

[0174] Following the previous example, suppose we obtain the position information Di1_pi of the first display sub-region in the display area and the position information Di2_pi of the second display sub-region in the display area.

[0175] Specifically, the grid layout information of the display screen is set to 10×10, and the position information of the gaze point is set to grid (5, 5). The first display sub-region can be a 5×5 grid, specifically grid (3, 3) and grid (7, 7), then the position information Di1_pi includes the position information of grid (3, 3) and grid (7, 7); correspondingly, the other grids outside of grid (3, 3) to grid (7, 7) are the second display sub-regions, then the position information Di2_pi includes the position information of grid (1, 1) and grid (10, 10).

[0176] S606, the position information of the first display sub-region is transformed based on the size information of the image to obtain the position transformation information of the first display sub-region, and the position information of the second display sub-region is transformed based on the size information of the image to obtain the position transformation information of the second display sub-region.

[0177] Following the previous example, by converting the location information Di1_pi, we can obtain the location conversion information Di1_pi' of the first display sub-area. Similarly, by converting the location information Di2_pi, we can obtain the location conversion information Di2_pi' of the second display sub-area.

[0178] Specifically, the coordinates of the starting grid of the first display sub-region (i.e., grid (3, 3)) are represented as follows: The coordinates of the terminating grid of the first display sub-region (i.e., grid (7, 7)) are represented as follows: ,in For the width of a single grid cell, This refers to the height of a single grid in pixels. For example, if the display resolution is 1920×1080 pixels, then the size of a single grid is 192×108 pixels. It is 192. The value is 108. Accordingly, the position transformation information Di1_pi' includes the position information of the starting grid. and the location information of the terminating grid. Similarly, the position transformation information Di2_pi' is obtained.

[0179] S607, based on the position transformation information of the first display sub-region, determine the first image sub-region corresponding to the first display sub-region from the image, and based on the position transformation information of the second display sub-region, determine the second image sub-region corresponding to the second display sub-region from the image.

[0180] Following the previous example, let the first image sub-region (also called the high-quality image region) determined using the position transformation information Di1_pi' be... ,in The dimensions of the first image sub-region are: and the dimensions of the second image sub-region (also known as the high-performance image region) are determined using the position transformation information Di2_pi'. ,in This represents the size of the second image sub-region.

[0181] S608, render the first image sub-region using a first resolution to obtain the rendered region image corresponding to the first image sub-region, and render the second image sub-region using a second resolution to obtain the rendered region image corresponding to the second image sub-region.

[0182] Optionally, a first region image (also known as a high-quality image) is generated by cropping a first image sub-region from the image using a first rendering model and rendering it based on a first resolution and the pixel values ​​contained in the first region image.

[0183] Following the previous example, let the first resolution be... The first rendering model (also known as the high-quality model or high-quality channel) uses the first resolution. For high-quality image regions Perform over-resolution processing, i.e. This generates the first rendering area image. .

[0184] Optionally, the pixel values ​​contained in the first image sub-region are determined from the image using the second rendering model, and the determined pixel values ​​are set to zero to obtain the second region image corresponding to the second image sub-region. The second rendering region image (also known as the high-performance image) is generated by rendering based on the second resolution and the pixel values ​​contained in the second region image.

[0185] Following the previous example, let the second resolution be... The second rendering model (also known as the high-performance model or high-performance channel) uses the second resolution. For high-performance image regions Perform over-resolution processing, i.e. This generates the second rendering area image. .

[0186] Understandably, in the aforementioned examples > .

[0187] S609, the rendering area image corresponding to the first image sub-region and the rendering area image corresponding to the second image sub-region are fused to obtain the rendered image.

[0188] Following the previous example, for high-quality images and high-performance images By merging the components, the rendered image can be obtained.

[0189] For better understanding, please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating an image rendering method according to an embodiment of this application. Figure 7 As shown, an image from the extended reality environment is input, and the gaze point is calculated based on the image. The image is then input into the high-quality channel and the high-performance channel. The high-quality image region is identified in the high-quality channel, and a super-resolution high-quality image is obtained. The high-performance image region is identified in the high-performance channel, and a super-resolution high-performance image is obtained. The high-quality channel and the high-performance channel perform super-resolution processing in parallel. Then, the super-resolution high-quality image and the super-resolution high-performance image are fused to obtain the super-resolution image (i.e., the rendered image), which is then output and displayed.

[0190] In this embodiment, the image in the extended reality environment is divided into a high-quality image region and a high-performance image region by using foveation information. Then, higher super-resolution processing is performed on the high-quality image region and lower super-resolution processing is performed on the high-performance image region. This achieves targeted rendering of images in the extended reality environment, improves rendering accuracy and efficiency, and ensures the reliability of image rendering.

[0191] Figure 8 This is a block diagram illustrating an image rendering apparatus according to an embodiment of this application. Figure 8 As shown, the device includes:

[0192] The acquisition module 801 is configured to acquire gaze point information of the carrier corresponding to the image in the extended reality environment.

[0193] The segmentation module 802 is configured to segment the display area of ​​the carrier based on the gaze point information to obtain multiple display sub-regions;

[0194] The conversion module 803 is configured to perform conversion based on the plurality of display sub-regions to obtain a plurality of image sub-regions for the image;

[0195] The rendering module 804 is configured to render the multiple image sub-regions using different rendering strategies to obtain a rendered image.

[0196] In one embodiment of this application, based on the foregoing scheme, the gaze point information includes the position information of the gaze point in the display area of ​​the carrier; the segmentation module 802 is specifically configured as follows:

[0197] Obtain the grid layout information of the grids contained in the display area;

[0198] The display area is divided based on the grid layout information and the position information of the gaze point to obtain multiple display sub-regions.

[0199] In one embodiment of this application, based on the foregoing scheme, the plurality of display sub-regions include a first display sub-region and a second display sub-region; the partitioning module 802 is further specifically configured as follows:

[0200] Based on the grid layout information, a first range area surrounding the position represented by the position information of the gaze point is selected from the display area;

[0201] The first range area is divided into the first display sub-region, and the other areas in the display area other than the first display sub-region are divided into the second display sub-region.

[0202] In one embodiment of this application, based on the foregoing scheme, the conversion module 803 is specifically configured as follows:

[0203] Obtain the position information of each display sub-region within the display area;

[0204] Based on the size information of the image, the position information of each display sub-region is transformed to obtain the position transformation information of each display sub-region;

[0205] Based on the position transformation information of each display sub-region, the image sub-region corresponding to each display sub-region is determined from the image.

[0206] In one embodiment of this application, based on the foregoing scheme, the location information of each display sub-region includes the location information of multiple display points located on the boundary of each display sub-region; the conversion module 803 is further specifically configured as follows:

[0207] Obtain the position information of multiple display points corresponding to the same display sub-region;

[0208] Based on the size information of the image, the position information of each display point is transformed to obtain the position transformation information of the same display sub-region.

[0209] In one embodiment of this application, based on the foregoing scheme, the conversion module 803 is further configured as follows:

[0210] If the plurality of display sub-regions includes a first display sub-region and a second display sub-region, and the second display sub-region is another region in the display region besides the first display sub-region, then the position information of the first display sub-region is transformed based on the size information of the image to obtain the position transformation information of the first display sub-region, and the position transformation information of the second display sub-region is obtained based on the position transformation information of the first display sub-region.

[0211] In one embodiment of this application, based on the foregoing scheme, the rendering module 804 is specifically configured as follows:

[0212] Render the multiple image sub-regions at different resolutions to generate a rendering area image corresponding to each image sub-region;

[0213] Multiple rendering region images are merged to obtain a rendered image.

[0214] In one embodiment of this application, based on the foregoing scheme, the plurality of image sub-regions include a first image sub-region and a second image sub-region; the rendering module 804 is further specifically configured as follows:

[0215] The image is input into a first rendering model, and the first image sub-region is determined from the image using the first rendering model. The first image sub-region is then rendered using a first resolution to generate a first rendering region image corresponding to the first image sub-region; and

[0216] The image is input into the second rendering model, and the second image sub-region is determined from the image through the second rendering model. The second image sub-region is rendered using the second resolution to generate the second rendering region image corresponding to the second image sub-region.

[0217] In one embodiment of this application, based on the foregoing scheme, the rendering module 804 is further configured as follows:

[0218] The first rendering model is used to crop out the first region image corresponding to the first image sub-region from the image, and the first rendering region image is generated by rendering based on the first resolution and the pixel values ​​contained in the first region image.

[0219] In one embodiment of this application, based on the foregoing scheme, the rendering module 804 is further configured as follows:

[0220] The second rendering model determines the pixel values ​​contained in the first image sub-region from the image, and sets the determined pixel values ​​to zero to obtain the second region image corresponding to the second image sub-region. The second rendering region image is generated by rendering based on the second resolution and the pixel values ​​contained in the second region image.

[0221] In one embodiment of this application, based on the foregoing scheme, the rendering module 804 is specifically configured as follows:

[0222] Based on the plurality of image sub-regions, an image sub-region group is obtained, wherein the image sub-region group includes two adjacent image sub-regions;

[0223] Render two image sub-regions in the image sub-region group using different rendering strategies to generate a rendering region image corresponding to each image sub-region in the image sub-region group;

[0224] Obtain the adjacent image regions of two image sub-regions in the image sub-region group, and generate the rendering region image corresponding to the adjacent image regions;

[0225] The rendered region images corresponding to the adjacent image regions and the rendered region images corresponding to each image sub-region in the image sub-region group are fused to obtain a rendered image.

[0226] In one embodiment of this application, based on the foregoing scheme, the rendering module 804 is further configured as follows:

[0227] Obtain a rendering strategy for rendering two image sub-regions in the image sub-region group respectively;

[0228] A range of rendering parameter values ​​is generated based on the rendering parameter values ​​corresponding to the two rendering strategies, and a target rendering parameter value located within the range of rendering parameter values ​​is generated.

[0229] The adjacent image regions are rendered using the target rendering parameter values ​​to generate rendering region images corresponding to the adjacent image regions.

[0230] In one embodiment of this application, based on the foregoing scheme, the rendering module 804 is further configured as follows:

[0231] From the rendered region image corresponding to each image sub-region in the image sub-region group, extract the candidate region image corresponding to the adjacent image region;

[0232] The rendering region image corresponding to the adjacent image region is generated by calculating based on two candidate region images and the weight value corresponding to each candidate region image.

[0233] It should be noted that the apparatus provided in the foregoing embodiments and the method provided in the foregoing embodiments belong to the same concept, and the specific way in which each module and unit performs operations has been described in detail in the method embodiments.

[0234] Embodiments of this application also provide an electronic device, including: one or more processors; and a memory for storing one or more computer programs, which, when executed by one or more processors, cause the electronic device to implement the aforementioned image rendering method.

[0235] Figure 9 It is an electronic device suitable for implementing the embodiments of this application (e.g. Figure 1 The diagram shows the structure of a computer system (terminal device or server).

[0236] It should be noted that, Figure 9 The computer system 900 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0237] like Figure 9As shown, the computer system 900 includes a Central Processing Unit (CPU) 901, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on a computer program stored in a Read-Only Memory (ROM) 902 or a computer program loaded from a storage portion 908 into a Random Access Memory (RAM) 903. The RAM 903 also stores various computer programs and data required for system operation. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An Input / Output (I / O) interface 905 is also connected to the bus 904.

[0238] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. Removable media 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 910 as needed so that computer programs read from them can be installed into storage section 908 as needed.

[0239] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as a computer program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer instructions for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs various functions defined in the system of this application.

[0240] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. For example, a computer-readable medium can be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a computer program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0241] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0242] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0243] Another aspect of this application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the image rendering method as described above. This computer-readable medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0244] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable medium. A processor of an electronic device reads the computer instructions from the computer-readable medium and executes the computer instructions, causing the electronic device to perform the image rendering methods provided in the various embodiments described above.

[0245] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. An image rendering method, characterized by, include: Obtain gaze point information of the carrier corresponding to the image in the extended reality environment; The display area of ​​the carrier is divided based on the gaze point information to obtain multiple display sub-regions; Based on the conversion of the multiple display sub-regions, multiple image sub-regions for the image are obtained; Rendering is performed on the multiple image sub-regions using different rendering strategies to obtain a rendered image; The step of converting based on the plurality of display sub-regions to obtain a plurality of image sub-regions for the image includes: Obtain the position information of each display sub-region within the display area; Based on the size information of the image, the position information of each display sub-region is transformed into the coordinate system of the extended reality environment to obtain the position transformation information of each display sub-region; Based on the position transformation information of each display sub-region, determine the image sub-region corresponding to each display sub-region from the image; The step of obtaining the position information of each display sub-region in the display area includes: The starting grid coordinates and ending grid coordinates are obtained from the boundary of the same display sub-region, and the starting grid coordinates and ending grid coordinates are combined to obtain the position information of the same display sub-region in the display area; The step of converting the position information of each display sub-region to the coordinate system of the extended reality environment based on the size information of the image to obtain the position transformation information of each display sub-region includes: Based on the resolution and grid layout of the display area, calculate the width and height of a single grid. Based on the width and the height, the starting grid coordinates and the ending grid coordinates corresponding to the same display sub-region are converted into pixel coordinates in the display region respectively; Based on the size information of the image, the pixel coordinates corresponding to the starting grid coordinates and the ending grid coordinates are normalized to the coordinate system of the extended reality environment to obtain the position transformation information of the same display sub-region.

2. The method of claim 1, wherein, The gaze point information includes the position information of the gaze point in the display area of ​​the carrier; the division of the display area of ​​the carrier based on the gaze point information to obtain multiple display sub-regions includes: Obtain the grid layout information of the grids contained in the display area; The display area is divided based on the grid layout information and the position information of the gaze point to obtain multiple display sub-regions.

3. The method of claim 2, wherein, The plurality of display sub-regions includes a first display sub-region and a second display sub-region; the division of the display area based on the grid layout information and the position information of the gaze point yields a plurality of display sub-regions, including: Based on the grid layout information, a first range area surrounding the position represented by the position information of the gaze point is selected from the display area; The first range area is divided into the first display sub-region, and the other areas in the display area other than the first display sub-region are divided into the second display sub-region.

4. The method of claim 1, wherein, The step of transforming the position information of each display sub-region based on the size information of the image to obtain the position transformation information of each display sub-region includes: If the plurality of display sub-regions includes a first display sub-region and a second display sub-region, and the second display sub-region is another region in the display region besides the first display sub-region, then the position information of the first display sub-region is transformed based on the size information of the image to obtain the position transformation information of the first display sub-region, and the position transformation information of the second display sub-region is obtained based on the position transformation information of the first display sub-region.

5. The method of claim 1, wherein, The rendering strategy includes resolution; the rendering of the multiple image sub-regions using different rendering strategies to obtain a rendered image includes: Render the multiple image sub-regions at different resolutions to generate a rendering area image corresponding to each image sub-region; Multiple rendering region images are merged to obtain a rendered image.

6. The method of claim 5, wherein, The plurality of image sub-regions includes a first image sub-region and a second image sub-region; the step of rendering the plurality of image sub-regions at different resolutions to generate a rendered region image corresponding to each image sub-region includes: The image is input into a first rendering model, and the first image sub-region is determined from the image using the first rendering model. The first image sub-region is then rendered using a first resolution to generate a first rendering region image corresponding to the first image sub-region; and The image is input into the second rendering model, and the second image sub-region is determined from the image through the second rendering model. The second image sub-region is rendered using the second resolution to generate the second rendering region image corresponding to the second image sub-region.

7. The method of claim 6, wherein, The step of determining the first image sub-region from the image using the first rendering model, rendering the first image sub-region using a first resolution, and generating a first rendering region image corresponding to the first image sub-region includes: The first region image corresponding to the first image sub-region is cropped from the image using the first rendering model, and the first rendering region image is generated by rendering based on the first resolution and the pixel values ​​contained in the first region image. The step of determining the second image sub-region from the image using the second rendering model, rendering the second image sub-region using the second resolution, and generating a second rendering region image corresponding to the second image sub-region includes: The second rendering model determines the pixel values ​​contained in the first image sub-region from the image, and sets the determined pixel values ​​to zero to obtain the second region image corresponding to the second image sub-region. The second rendering region image is generated by rendering based on the second resolution and the pixel values ​​contained in the second region image.

8. The method according to any one of claims 1 to 7, characterized in that, The rendering of the multiple image sub-regions using different rendering strategies to obtain a rendered image includes: Based on the plurality of image sub-regions, an image sub-region group is obtained, wherein the image sub-region group includes two adjacent image sub-regions; Render two image sub-regions in the image sub-region group using different rendering strategies to generate a rendering region image corresponding to each image sub-region in the image sub-region group; Obtain the adjacent image regions of two image sub-regions in the image sub-region group, and generate the rendering region image corresponding to the adjacent image regions; The rendered region images corresponding to the adjacent image regions and the rendered region images corresponding to each image sub-region in the image sub-region group are fused to obtain a rendered image.

9. The method of claim 8, wherein, The process of generating the rendering region image corresponding to the adjacent image region includes: Obtain a rendering strategy for rendering two image sub-regions in the image sub-region group respectively; A range of rendering parameter values ​​is generated based on the rendering parameter values ​​corresponding to the two rendering strategies, and a target rendering parameter value located within the range of rendering parameter values ​​is generated. The adjacent image regions are rendered using the target rendering parameter values ​​to generate rendering region images corresponding to the adjacent image regions.

10. The method of claim 8, wherein, The process of generating the rendering region image corresponding to the adjacent image region includes: From the rendered region image corresponding to each image sub-region in the image sub-region group, extract the candidate region image corresponding to the adjacent image region; The rendering region image corresponding to the adjacent image region is generated by calculating based on two candidate region images and the weight value corresponding to each candidate region image.

11. An image rendering apparatus, characterized by comprising: include: The acquisition module is configured to acquire gaze point information of the carrier corresponding to the image in the extended reality environment. The segmentation module is configured to segment the display area of ​​the carrier based on the gaze point information to obtain multiple display sub-regions; The conversion module is configured to perform conversion based on the multiple display sub-regions to obtain multiple image sub-regions for the image; The rendering module is configured to render the multiple image sub-regions using different rendering strategies to obtain a rendered image; The step of converting based on the plurality of display sub-regions to obtain a plurality of image sub-regions for the image includes: Obtain the position information of each display sub-region within the display area; Based on the size information of the image, the position information of each display sub-region is transformed into the coordinate system of the extended reality environment to obtain the position transformation information of each display sub-region; Based on the position transformation information of each display sub-region, determine the image sub-region corresponding to each display sub-region from the image; The step of obtaining the position information of each display sub-region in the display area includes: The starting grid coordinates and ending grid coordinates are obtained from the boundary of the same display sub-region, and the starting grid coordinates and ending grid coordinates are combined to obtain the position information of the same display sub-region in the display area; The step of converting the position information of each display sub-region to the coordinate system of the extended reality environment based on the size information of the image to obtain the position transformation information of each display sub-region includes: Based on the resolution and grid layout of the display area, calculate the width and height of a single grid. Based on the width and the height, the starting grid coordinates and the ending grid coordinates corresponding to the same display sub-region are converted into pixel coordinates in the display region respectively; Based on the size information, the pixel coordinates corresponding to the starting grid coordinates and the ending grid coordinates are normalized to the coordinate system of the extended reality environment to obtain the position transformation information of the same display sub-region.

12. The apparatus of claim 11, wherein, The gaze point information includes the position information of the gaze point in the display area of ​​the carrier; the segmentation module is specifically configured as follows: Obtain the grid layout information of the grids contained in the display area; The display area is divided based on the grid layout information and the position information of the gaze point to obtain multiple display sub-regions.

13. The apparatus of claim 12, wherein, The plurality of display sub-regions include a first display sub-region and a second display sub-region; the partitioning module is further specifically configured as follows: Based on the grid layout information, a first range area surrounding the position represented by the position information of the gaze point is selected from the display area; The first range area is divided into the first display sub-region, and the other areas in the display area other than the first display sub-region are divided into the second display sub-region.

14. An electronic device, comprising: include: One or more processors; A memory for storing one or more computer programs that, when executed by the electronic device, cause the electronic device to implement the image rendering method as described in any one of claims 1 to 10.

15. A computer readable medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the image rendering method as described in any one of claims 1 to 10.

16. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the image rendering method as described in any one of claims 1 to 10.

Images