Foveated rendering method and device based on subspace structure, electronic device and storage medium

Abstract

The present invention provides a foveation rendering method and device based on a subspace structure, an electronic device and a storage medium. The method comprises: obtaining an original frame image to be processed, and dividing the original frame image into regions with different resolutions; performing downsampling processing on the regions with different resolutions respectively; performing subspace division on the downsampled image; encapsulating and transmitting the image data in the same subspace as a data block; the receiving end decapsulates the image data and stores it in a memory of a buffer; performing upsampling processing on the image data in the same subspace in the memory; performing blurring processing and boundary fusion processing on the regions with different resolutions in the same subspace respectively to obtain a subspace reconstructed image; transmitting several subspace reconstructed images to a display buffer in sequence, providing display data for a display driver and displaying them through a display panel. The present application can reduce the amount of data transmission and the data storage capacity of a display chip.

Description

Technical Field

[0001] The present application relates to the field of display technology, and in particular to a foveation rendering method and device based on a subspace structure, an electronic device and a storage medium. Background Art

[0002] Foveation Rendering (FR) is a new image rendering technology that simulates the visual processing of the human eye. It provides high-resolution rendering effects for the foveation area, and low-resolution rendering effects for the area outside the foveation area. This method selectively renders the clear range of the human eye's field of view, which helps reduce the computing load of the GPU and also reduces the data transmission bandwidth between the GPU and the display chip.

[0003] However, the current mainstream practice in the industry is still to use a method of transmitting image data of different resolution areas separately. For example, in patent US10949947B2, the image is divided into three types of areas: a high-resolution gaze area, a medium-resolution transition area, and a peripheral low-resolution area. The high-resolution data of the left and right eye gaze areas are transmitted first, then the medium-resolution data of the transition area is transmitted, and finally the data of the peripheral low-resolution area is transmitted. This results in the display chip having to store the data of the high-resolution gaze area and the middle transition area first. After that, when the data of the peripheral low-resolution area is received, it can be combined with the data of the other two areas previously stored to fuse the entire image for display, which undoubtedly increases the data storage capacity of the display chip. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present application provides a foveated rendering method based on a subspace structure, an electronic device, a storage medium and a program product.

[0005] The present application provides a foveated rendering method based on a subspace structure, comprising the steps of: obtaining an original frame image to be processed, and dividing the original frame image into regions with different resolutions; performing downsampling processing on the regions with different resolutions respectively; performing subspace division on the downsampled image; encapsulating and transmitting image data in the same subspace as a data block; decapsulating the image data at a receiving end and storing it in a memory of a buffer; performing upsampling processing on the image data in the same subspace in the memory; performing blurring processing and boundary fusion processing on regions with different resolutions in the same subspace respectively to obtain a subspace reconstructed image; transmitting a plurality of subspace reconstructed images to a display buffer in sequence, providing display data for a display driver, and displaying them through a display panel.

[0006] Optionally, dividing the original frame image into different resolution areas includes the steps of: dividing the original frame image into m types of different resolution areas according to division information, m≥2; wherein the division information includes gaze point position information and size information of different resolution areas, wherein the gaze point position information includes static gaze point position information or dynamic gaze point position information; the different resolution areas include a gaze area and m-1 types of areas other than the gaze area.

[0007] Optionally, the m-1 type of areas other than the attention area include: a transition area and a non-attention area, wherein the transition area is located between the attention area and the non-attention area.

[0008] Optionally, the downsampling processing for the different resolution areas respectively includes the steps of: presetting different scaling ratios for the different resolution areas; and downsampling the different resolution areas respectively according to the preset different scaling ratios.

[0009] Optionally, the subspace division of the downsampled image includes the steps of: obtaining the inverse of the scaling ratio of each different resolution area in the selected direction; obtaining the least common multiple of the inverse of all scaling ratios, and dividing the subspace using it as a division criterion.

[0010] Optionally, the foveation rendering method based on the subspace structure also includes the steps of: in a selected direction, determining whether the boundary of two adjacent areas of different resolutions is located at the subspace boundary; if so, encapsulating the image data in the same subspace as a data block; if not, adjusting the boundary of the two adjacent areas of different resolutions to align with the boundary of the subspace.

[0011] Optionally, the up-sampling processing of the image data in the same subspace in the memory includes the steps of: using an up-sampling algorithm to up-sample the image data in different resolution areas in the subspace respectively.

[0012] The present application also provides a foveation rendering device based on a subspace structure, the device comprising: a region division module, used to obtain an original frame image to be processed and division information, and divide the original frame image into regions with different resolutions according to the division information; a downsampling module, used to perform downsampling processing on regions with different resolutions according to the scaling ratios of regions with different resolutions; a subspace division module, used to perform subspace division; a data block encapsulation module, used to encapsulate image data in the same subspace as a data block; a data sending module, used to transmit the encapsulated image data; a data receiving module, used to receive image data transmitted from the data sending module; an upsampling module, used to perform upsampling processing on regions with different resolutions in the same subspace; a blur processing module, used to perform blur processing on regions with different resolutions in the same subspace; and a boundary fusion module, used to perform boundary fusion processing on regions with different resolutions in the same subspace.

[0013] The present application also provides an electronic device, comprising: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above-mentioned subspace structure-based foveated rendering method.

[0014] The present application also provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the above-mentioned foveated point rendering method based on the subspace structure is implemented.

[0015] Compared with the prior art, the technical solution of the present application has the following beneficial effects:

[0016] The present application provides a foveated rendering method and device, an electronic device and a storage medium based on a subspace structure. By dividing image data into multiple subspaces according to a preset scaling ratio, the subspace is transmitted as a minimum data transmission unit. During the display process, it is possible to reconstruct the image while receiving subspace data packets, and render different display areas with different resolutions. On the one hand, the amount of data transmission is reduced, and on the other hand, the data storage capacity of the display chip is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1A schematic diagram of a flow chart of a foveated rendering method based on a subspace structure according to an embodiment of the present application;

[0019] Figure 2 This is a structural schematic diagram of a foveated rendering device based on a subspace structure according to an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of an image division method in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of a downsampling method according to a first embodiment of the present application;

[0022] Figure 5 A schematic diagram of a downsampling method according to a second embodiment of the present application;

[0023] Figure 6 This is a schematic diagram of subspace division in the first embodiment of the present application;

[0024] Figure 7 This is a schematic diagram of subspace division according to the second embodiment of the present application;

[0025] Figure 8 This is a schematic diagram of a structure in which the boundaries of different resolution areas are located inside a subspace according to an embodiment of the present application;

[0026] Fig. 9 This is a structural schematic diagram of an embodiment of the present application in which the boundary of different resolution areas is located at the subspace boundary;

[0027] Fig.10 This is a schematic diagram of the structure of a subspace in one embodiment of the present application;

[0028] Figure 11-13 Schematic diagram of three different encapsulation and transmission modes of a data encapsulation process in an embodiment of the present application;

[0029] Fig.14 This is a schematic diagram of the matching between the data receiving module and the data sending module in one embodiment of the present application;

[0030] Figure 15-16 A schematic diagram of the process of the upsampling method of the first embodiment of the present application;

[0031] Figure 17-Figure 18 A schematic diagram of the process of the upsampling method according to the second embodiment of the present application;

[0032] Figure 19-Figure 23 This is a schematic diagram of the process of the upsampling method according to the third embodiment of the present application. DETAILED DESCRIPTION

[0033] The preferred embodiments of the present application will be described in more detail below. Although the preferred embodiments of the present application are described below, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein.

[0034] In the present application, unless otherwise specified, the directional words used, such as "upper and lower", generally refer to the upper and lower parts of the device in normal use, and "inside and outside" refer to the outline of the device. In addition, the terms "first, second, third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, features defined as "first, second, third" may explicitly or implicitly include one or more of the features. In the description of the present application, "multiple" means two or more, unless otherwise clearly and specifically defined. Since the accompanying drawings are descriptions of the same device, the same reference numerals in the drawings represent the same components.

[0035] As described in the background technology, currently, in the foveated rendering technology, the image data is mostly transmitted in a transmission mode of transmitting data of different resolution areas separately, which results in the display chip having to store the first transmitted data before it can be fused and displayed, thereby increasing the storage capacity. Therefore, the present application provides a foveated rendering method and device based on a subspace structure, an electronic device and a storage medium, which can render different display areas with different resolutions, reducing the amount of data transmission on the one hand, and reducing the data storage capacity of the display chip on the other hand.

[0036] The present application provides a foveated rendering method based on a subspace structure, comprising the steps of: obtaining an original frame image to be processed, and dividing the original frame image into regions with different resolutions; performing downsampling processing on the regions with different resolutions respectively; performing subspace division on the downsampled image; encapsulating and transmitting image data in the same subspace as a data block; decapsulating the image data at a receiving end and storing it in a memory of a buffer; performing upsampling processing on the image data in the same subspace in the memory; performing blurring processing and boundary fusion processing on regions with different resolutions in the same subspace respectively to obtain a subspace reconstructed image; transmitting a plurality of subspace reconstructed images to a display buffer in sequence, providing display data for a display driver, and displaying them through a display panel.

[0037] Furthermore, the present application also provides a foveation rendering device based on a subspace structure, the device comprising: a region division module, used to obtain an original frame image to be processed and division information, and divide the original frame image into regions with different resolutions according to the division information; a downsampling module, used to downsample regions with different resolutions according to scaling ratios of regions with different resolutions; a subspace division module, used to perform subspace division; a data block encapsulation module, used to encapsulate image data within the same subspace as a data block; a data sending module, used to transmit the encapsulated image data; a data receiving module, used to receive image data transmitted from the data sending module; an upsampling module, used to upsample regions with different resolutions within the same subspace; a blur processing module, used to blur regions with different resolutions within the same subspace; and a boundary fusion module, used to perform boundary fusion on regions with different resolutions within the same subspace.

[0038] In order to provide a further understanding of the purpose, structure, features and functions of the present application, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0039] Reference Figure 1 , the present application provides a foveated rendering method based on a subspace structure, comprising the steps of:

[0040] S1: Acquire an original frame image to be processed, and divide the original frame image into regions with different resolutions.

[0041] In the embodiments of the present application, Figure 2 As shown, the region division module in the subspace structure-based foveated rendering device divides the original frame image into m types of regions with different resolutions according to the division information, where m≥2. The division information includes foveated point position information, size information of regions with different resolutions, etc., where the foveated point position information includes static foveated point position information or dynamic foveated point position information.

[0042] In some embodiments, the static gaze point position may be a pre-set position, for example, the gaze point is fixed at the center of the image; the dynamic gaze point position information is the dynamic gaze point position information obtained by acquiring the movement information of the human eye in real time using eye tracking technology and performing a series of calculations.

[0043] In different embodiments, the division information used to divide the original frame image may be different. For example, the original frame image may be divided into different resolution areas according to real-time human eye gaze point position information or preset fixed gaze point position information, or preset size information of two or more different resolution area types.

[0044] In some embodiments, the size information refers to the height and width of different resolution areas, which can be preset as a fixed value or set as a percentage of the height and width of the original resolution image. The height and width of different areas can be flexibly adjusted according to actual applications. The size information of two or more different resolution area types means that different resolution area types can be divided using two or more size information, and more types of areas can be set according to actual conditions, and this application does not limit this.

[0045] Specifically, in one embodiment of the present application, Figure 3 As shown, the region division module divides the entire image into three types of regions based on the center position of the gaze point of the original frame image: a gaze area, a transition area, and a non-gaze area.

[0046] In some embodiments, the m types of regions with different resolutions may include two types of regions: a fixation region and a non-fixation region, or may include a fixation region, a transition region, and a non-fixation region; or may include multiple types of regions with different resolutions, such as a fixation region, a first transition region, a second transition region, and a non-fixation region. In different embodiments, the number m of different resolution region types may have different values, and this application does not limit this.

[0047] S2: Downsample the regions with different resolutions separately.

[0048] In an embodiment of the present application, the downsampling processing of different resolution areas respectively includes the steps of: presetting different scaling ratios for each different resolution area; and downsampling the original image data by the downsampling module according to the preset different scaling ratios.

[0049] Scaling ratio is the parameter for downsampling different resolution areas. Figure 4 In the first embodiment of the present application, the different resolution areas are downsampled respectively by the downsampling module to retain the original resolution of the attention area; for the transition area, the horizontal and vertical directions are downsampled at a ratio of 1 / 2, and after downsampling, the data in this area is only 1 / 4 of the original data; for the surrounding non-attention areas, the horizontal and vertical scaling ratios are downsampled at a ratio of 1 / 4, and after downsampling, the data in this area is only 1 / 16 of the original data.

[0050] In the second embodiment of the present application, refer to Figure 5 , you can also use Figure 4 The embodiments shown show different downsampling methods. At the same time, it should be noted that in different embodiments, the downsampling scaling ratios in the horizontal and vertical directions of the image may be the same or different.

[0051] S3: Perform subspace division on the downsampled image.

[0052] The direction in which the subspace is divided corresponds to the image scanning direction. In different embodiments, the subspace can be divided by row or by column. In some embodiments, since the image is displayed by line scanning, the image is divided into multiple subspaces in the row direction. That is, according to the scaling ratio in the vertical direction, the number of rows of each subspace is determined, and the downsampled image is divided according to the obtained number of rows of the subspace.

[0053] The subspace division of the downsampled image includes the steps of: obtaining the inverse of the scaling ratio of each area with different resolutions in the selected direction; obtaining the least common multiple of the inverse of all scaling ratios, and dividing the subspace using it as a division standard.

[0054] In some examples, the region division module divides the image into three region types: a fixation region, a transition region, and a non-fixation region. The downsampling module uses The scaling ratio is adopted in the transition area in the middle. The scaling ratio is used in the non-fixation area. The sampling is performed at the scaling ratio of . Therefore, the inverse of the scaling ratio in the vertical direction is , and . Thus, the number of subspace rows is the least common multiple of the reciprocals of the three scaling ratios. Afterwards, the subspace division module divides the downsampled image into is a subspace, divided into several subspaces.

[0055] For example, Figure 6 As shown, in the first embodiment of the present application, the original resolution is used in the fixation area and the transition area is used The scaling ratio of the non-fixation area is In the case of the scaling ratio, the reciprocals of the scaling ratios in the vertical direction are 1, 2, and 4, respectively, and their least common multiple is 4. Therefore, the downsampled image can be divided into several subspaces with every 4 rows as a subspace.

[0056] In the second embodiment of the present application, refer to Figure 7 , the same 4 lines of image data are used as a subspace, but due to the use of a different downsampling method from the first embodiment, the subspace obtained by division is different from Figure 6 The embodiments shown are different.

[0057] It should be noted that the foveated rendering method based on the subspace structure also includes the steps of: in the selected direction, determining whether the boundary of two adjacent regions with different resolutions is located at the subspace boundary; if so, encapsulating the image data in the same subspace as a data block; if not, adjusting the boundary of the two adjacent regions with different resolutions to align with the boundary of the subspace.

[0058] In some embodiments, due to the change of the gaze point, the boundary between regions of different resolutions may be located inside a certain subspace, such as Figure 8 As shown in FIG. 1 , subspace 1 contains the boundary of the non-attention area and the transition area. In this case, the boundary of the different resolution areas of the current frame can be adjusted to ensure that the boundary between the different resolution areas is not included in the subspace, that is, adjusted to Fig. 9 The situation shown.

[0059] S4: Encapsulate and transmit the image data in the same subspace as a data block.

[0060] Next, the downsampled data in the same subspace is taken as a data block and encapsulated according to a specific data transmission format to facilitate data transmission.

[0061] In different embodiments, different data encapsulation methods may be used. Specifically, Fig.10 In the 4-row subspace shown, the same row only contains downsampled pixel data, the focus area uses the original resolution, and the transition area uses The scaling ratio of the non-fixation area is For Fig.10 In the case where the same subspace shown contains data of multiple resolutions, the following three pixel data packaging and transmission methods may be used but are not limited to them.

[0062] During the data encapsulation process, the data encapsulation module and the data transmission module can encapsulate and transmit in a scanning order from left to right and from top to bottom. Fig.11 , which only includes the downsampled data part. In different embodiments, additional information can be added according to actual conditions.

[0063] In some other embodiments, the data encapsulation module and the data sending module may respectively encapsulate and transmit data according to different resolution types, and stipulate that the data of each resolution type is located in the order of the data packet. Fig.12 The downsampled data portion included therein is shown. If necessary, in different embodiments, additional information may be added to the data packet, including but not limited to: data length information for each resolution data.

[0064] In some other embodiments, Fig.13 As shown, data of different resolution areas in the same subspace can also be transmitted through different data transmission channels.

[0065] S5: The receiving end decapsulates the image data and stores it in the memory of the buffer.

[0066] The data receiving module obtains the data information of the subspace from the data transmission module, decapsulates it and stores it in the memory of the buffer, such as Fig.14 As shown, the data receiving module matches the data sending module, thereby ensuring that the received data is consistent.

[0067] In some embodiments, two memories may be used alternately when receiving the downsampled data of the subspace.

[0068] S6: Perform upsampling processing on the image data in the same subspace in the memory.

[0069] The up-sampling process for the image data in the same subspace in the memory comprises the steps of: using an up-sampling algorithm to respectively up-sample the image data in different resolution areas in the subspace.

[0070] Since there are areas with different resolutions in the same subspace, that is, the same subspace contains non-attention area data, transition area data and attention area data at the same time, the upsampling module restores the low-resolution area to the same resolution as the attention area by adopting a method matching the downsampling, so as to facilitate display on the display panel.

[0071] In some embodiments, the image data in the subspace may be upsampled using a neighboring interpolation method or a bilinear interpolation method; in other embodiments, other different upsampling algorithms may be used to upsample the image data, which is not limited in the present application.

[0072] In the first embodiment of the present application, due to the use of Figure 4 The downsampling method shown in the figure needs to rely on the first row of downsampled data of the next adjacent subspace when upsampling a subspace. Therefore, in the storage process described above, two memories with the same number of rows as the subspace are required to store the data of the subspace.

[0073] In the upsampling process of the first embodiment of the present application, if Fig.15As shown, first, the down-sampled data of subspace i is stored in memory 1; then, in the next cycle, the down-sampled data of subspace i+1 is received and stored in memory 2; at the same time, the image of subspace i is reconstructed by combining the down-sampled data of subspace i stored in memory 1 in the previous cycle and the down-sampled data of the first row of subspace i+1.

[0074] Next, the subspace i+1 is upsampled in the same way, as Fig.16 As shown, the downsampled data of subspace i+2 is first received and stored in memory 1, and then the image of subspace i+1 is reconstructed by an upsampling method in combination with the downsampled data of subspace i+1 stored in memory 2 in the previous cycle and the downsampled data of the first row of subspace i+2.

[0075] A similar method is then used to reconstruct the images of each subspace in turn by alternating between two memories, thereby achieving reconstruction of the entire frame of the image.

[0076] In the second embodiment of the present application, Figure 5 The downsampling method shown in the figure depends on the first row of downsampled data of the next adjacent subspace when upsampling a certain subspace. Therefore, in the storage process described above, three memories with the same number of rows as the subspace are required. At the same time, the use of three memories can realize the synchronization of data reception and upsampling process.

[0077] Specifically, in the upsampling process of the second embodiment of the present application, for the sake of generality, it is assumed that the reconstruction of the first i-1 subspaces has been completed and the reconstruction of subspace i is currently started, referring to Fig.17 , in the i-th cycle, the data receiving module receives the downsampled data of subspace i+2 and stores it in memory 3; at the same time, according to the downsampled data of subspace i in memory 1 and the downsampled data of the first row of subspace i+1 in memory 2, the image of subspace i is reconstructed by upsampling.

[0078] Next, refer to Fig.18 , in the i+1th cycle, the data receiving module receives the downsampled data of subspace i+3 and stores it in memory 1. At the same time, according to the downsampled data of subspace i+1 in memory 2 and the downsampled data of the first row of subspace i+2 in memory 3, the image of subspace i+1 is reconstructed by upsampling.

[0079] A similar method is then used to reconstruct the image of each subspace in turn by alternating three storage cycles, thereby achieving reconstruction of the entire frame image.

[0080] In the third embodiment of the present application, the same downsampling method as the second embodiment is adopted, that is, Figure 5 The downsampling method shown is used to obtain image data. In this case, the data of the subspace can also be stored in two row memories with one more row than the number of rows in the subspace, which can save more storage.

[0081] Specifically, for the sake of clarity and completeness, assume that the fixation point is Fig.19 As shown in , the upper boundary of the first subspace is the first row of the image to be processed. First, refer to Fig. 20 The data receiving module pre-receives the down-sampled data of subspace 0, which is the down-sampled data of rows 0-3 of the original image, and stores the data in rows 0 to 3 of memory 1.

[0082] In upsampling period 0, refer to Fig.21 The data receiving module receives the downsampled data of subspace 1, that is, the downsampled data of rows 4-7 of the original image, and stores the data in rows 0-3 of memory 2; at the same time, upsample and reconstruct subspace 0, and the upsampling method here includes but is not limited to the nearest neighbor interpolation method.

[0083] In upsampling period 1, refer to Fig. 22 The data receiving module receives the downsampled data of subspace 2, that is, the downsampled data of rows 8-11 of the original image, and stores the data starting from row 4 of memory 1. When the data is stored to the last row of the current memory, the data is continued to be stored starting from row 0 of memory 1. Finally, the downsampled data of subspace 2 is stored in rows 4, 0, 1, and 2 of memory 1. At the same time, according to the downsampled data of subspace 1 stored in rows 0-3 of memory 2, combined with the downsampled data of the last row of subspace 0 in row 3 of memory 1, the upsampled image of subspace 1 is reconstructed. The upsampling method here includes but is not limited to the bilinear interpolation method.

[0084] In upsampling period 2, refer to Fig.23 , the data receiving module receives the downsampled data of subspace 3, that is, the downsampled data of rows 12-15 of the original image, and stores the data starting from row 4 of memory 2. After storing to the last row of the current memory, the data continues to be stored from row 0. Finally, the downsampled data of subspace 3 is stored in rows 4, 0, 1, and 2 of memory 2. At the same time, according to the downsampled data of subspace 2 in rows 4, 0-2 of memory 1, and combined with the downsampled data of the last row of subspace 1 in row 3 of memory 2, image reconstruction is performed for subspace 2. The upsampling method here includes but is not limited to the bilinear interpolation method. Similarly, each subspace of the image to be processed is reconstructed by alternating between two memories to obtain a reconstructed complete image.

[0085] For some embodiments that use the same upsampling method as the third embodiment described above, the process of storing the subspace data and reconstructing the image through two line memories is shown in the following table, where the number of rows in each memory is one more than the number of rows in the subspace:

[0086]

[0087] S7: performing blurring processing and boundary fusion processing on regions with different resolutions in the same subspace respectively to obtain a subspace reconstructed image.

[0088] Since the areas other than the gaze area are downsampled at different ratios, the image information will be lost during the downsampling process. Therefore, even after upsampling, the image quality of these areas will be reduced, especially the aliasing phenomenon is unavoidable. Therefore, blurring is required. By applying different degrees of blurring to different areas, the aliasing phenomenon of the image can be reduced.

[0089] In addition, only through the processing of the upsampling module, different resolution areas can be restored to the same resolution, but there will be obvious differences in display quality at the boundaries of different resolution areas. Therefore, in order to ensure a natural transition, it is necessary to perform a certain boundary fusion transition on the boundaries between different resolution areas, so as to achieve the effect of removing the boundary effect and achieving a natural transition between different resolution areas.

[0090] S8: transmitting the plurality of subspace reconstructed images to the display buffer in sequence, providing display data for the display driver and displaying them through the display panel.

[0091] In an embodiment of the present application, after blurring and boundary fusion processing, the subspace reconstructed image is sequentially transmitted to a display buffer to provide display data for a display driver, and the image is displayed through a display panel.

[0092] The present application provides a foveated point rendering method based on a subspace structure, which divides image data into multiple subspaces according to a preset scaling ratio, and uses the subspace as the minimum data transmission unit. During the display process, it is possible to reconstruct the image while receiving subspace data packets, and use different resolutions for rendering in different display areas, which reduces the amount of data transmission on the one hand, and reduces the data storage capacity of the display chip on the other.

[0093] The present application also provides an electronic device, comprising: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the subspace structure-based foveated rendering method.

[0094] The present application also provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the foveated point rendering method based on the subspace structure is implemented.

[0095] It is understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the method and device for foveated rendering based on the subspace structure, the electronic device, and the storage medium. In other embodiments of the present application, more or fewer components than shown in the figure may be included, or some components may be combined, or some components may be split, or different component arrangements may be provided. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0096] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0097] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner.

[0098] Program code can be implemented with high-level programming language or target area-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0099] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including, but not limited to, floppy disks, optical disks, optical discs, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0100] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0101] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0102] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structural or equivalent process transformations made using the contents of the specification and drawings of the present application, such as the mutual combination of technical features between the embodiments, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A foveated rendering method based on subspace structure, characterized in that: Includes steps: Acquire an original frame image to be processed, and divide the original frame image into regions with different resolutions; Downsample different resolution areas separately; Perform subspace division on the downsampled image; Encapsulate and transmit image data in the same subspace as a data block; The receiving end decapsulates the image data and stores it in the memory of the buffer; Perform upsampling processing on the image data in the same subspace in the memory; The different resolution areas in the same subspace are subjected to blurring and boundary fusion processing respectively to obtain the subspace reconstructed image; The plurality of subspace reconstructed images are sequentially transmitted to a display buffer, providing display data for a display driver and displaying the images through a display panel; The subspace division of the downsampled image comprises the steps of: In the selected direction, obtain the inverse of the scaling ratio of each area with different resolutions; The least common multiple of the reciprocals of all scaling factors is obtained and used as a partitioning criterion to partition the subspace.

2. The foveated rendering method based on subspace structure according to claim 1, characterized in that: The step of dividing the original frame image into different resolution areas comprises the following steps: According to the segmentation information, the original frame image is divided into m types of regions with different resolutions, m ≥ 2; among them, The division information includes gaze point position information and size information of different resolution areas, wherein the gaze point position information includes static gaze point position information or dynamic gaze point position information; The m types of regions with different resolutions include a fixation area and m-1 types of regions other than the fixation area.

3. The foveated rendering method based on subspace structure according to claim 2, characterized in that: The m-1 types of regions other than the attention area include: a transition area and a non-attention area, wherein the transition area is located between the attention area and the non-attention area.

4. The foveated rendering method based on subspace structure according to claim 3, characterized in that: The downsampling process for the different resolution areas comprises the following steps: Preset different zoom ratios for different resolution areas; According to different preset scaling ratios, downsampling is performed on different resolution areas respectively.

5. The foveated rendering method based on subspace structure according to claim 3, characterized in that: Also includes the steps: In the selected direction, determine whether the boundary of two adjacent regions with different resolutions is located at the subspace boundary; If so, the image data in the same subspace is encapsulated as a data block; if not, the boundaries of two adjacent regions with different resolutions are adjusted to align with the boundary of the subspace.

6. The foveated rendering method based on subspace structure according to claim 1, characterized in that: The up-sampling process of the image data in the same subspace in the memory comprises the steps of: An upsampling algorithm is used to perform upsampling processing on the image data of different resolution areas in the subspace.

7. A foveated rendering device based on a subspace structure using the foveated rendering method according to claim 1, characterized in that: The device comprises: A region division module, used for obtaining an original frame image to be processed and division information, and dividing the original frame image into regions with different resolutions according to the division information; A downsampling module is used to downsample different resolution areas according to their scaling ratios; A subspace partitioning module, used for partitioning subspaces; A data block encapsulation module is used to encapsulate image data in the same subspace as a data block; A data sending module, used for transmitting the encapsulated image data; A data receiving module, used for receiving the image data transmitted from the data sending module; An upsampling module is used to perform upsampling processing on regions with different resolutions in the same subspace; A blur processing module is used to blur areas with different resolutions in the same subspace; The boundary fusion module is used to perform boundary fusion processing on regions with different resolutions in the same subspace.

8. An electronic device, characterized in that: include: one or more processors; a memory for storing executable instructions; The one or more processors are configured to call the executable instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images