Image processing method, device and storage medium

Abstract

An image processing method, device and storage medium relate to the field of image processing technology, and can reduce the processing pressure of the CPU as much as possible while meeting the user's demand for high-definition content. The method includes: an electronic device obtains the user's pupil coordinates, and determines the user's gaze area and non-gaze area on the display according to the pupil coordinates; the electronic device uses a high resolution to render the image of the gaze area, and uses a low resolution to render the image of the non-gaze area, thereby reducing the rendering pressure of the GPU of the electronic device. In addition, the electronic device can also splice the rendered image of the gaze area and the rendered image of the non-gaze area, and display the spliced ​​image on the display. Since the image of the gaze area in the spliced ​​image is a high-definition image, the user's demand for high-definition images is met.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device and storage medium. Background Art

[0002] With the development of image processing technology, people have higher and higher requirements for the display effects of electronic devices. In order to meet the needs of users, manufacturers have begun to pursue higher-quality display effects. For example, they have adopted screens that support high refresh rates to achieve more coherent and smooth dynamic displays.

[0003] Normally, electronic devices render each frame of an image at full resolution and display the rendered image. However, the bandwidth transmission capacity of electronic devices and the rendering capacity of the graphics processing unit (GPU) / graphics card are limited. If the image is rendered at full resolution, the frame rate often cannot match the refresh rate of the display, and the processing pressure of the GPU is increased. Therefore, while meeting the user's demand for high-definition content, reducing the processing pressure of the GPU as much as possible has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides an image processing method, device and storage medium for reducing the processing pressure of a GPU as much as possible while satisfying the user's demand for high-definition content.

[0005] On the one hand, an image processing method is provided, which is applied to an electronic device with a display, and the method includes: the electronic device obtains the pupil coordinates of the user at the current moment, and determines the gaze area and non-gaze area corresponding to the display at the current moment according to the pupil coordinates of the user; the electronic device renders the image corresponding to the gaze area according to a first resolution, and renders the image corresponding to the non-gaze area according to a second resolution, and the first resolution is greater than the second resolution; the electronic device splices the rendered image corresponding to the gaze area with the rendered image corresponding to the non-gaze area, and displays the spliced ​​image on the display.

[0006] Based on the above technical solution, after obtaining the pupil coordinates of the user, the electronic device can determine the user's gaze area and non-gaze area on the display according to the pupil coordinates. Furthermore, the electronic device can use high resolution to render the image of the gaze area, and use low resolution to render the image of the non-gaze area, that is, the electronic device can use different resolutions to render the same frame of image. Compared with using high resolution to render the complete image, the rendering pressure of the GPU of the electronic device is reduced. In addition, the electronic device can also splice the rendered image of the gaze area and the rendered image of the non-gaze area, and display the spliced ​​image on the display. Since the image of the gaze area in the spliced ​​image is a high-definition image, the user's demand for high-definition images is met.

[0007] In some embodiments, the non-attention area includes a differential area and a low-definition area, the differential area is the area of ​​the attention area corresponding to the display at the first moment except the first area, the first area is the overlapping area of ​​the attention area corresponding to the display at the first moment and the attention area corresponding to the display at the current moment, and the first moment is the previous moment of the current moment; the low-definition area is the area of ​​the display except the attention area at the first moment and the attention area at the current moment; the method also includes: rendering the image corresponding to the attention area at the current moment and the image corresponding to the differential area according to the first refresh rate, and rendering the image corresponding to the low-definition area according to the second refresh rate, and the first refresh rate is greater than the second refresh rate.

[0008] Based on this possible embodiment, the electronic device can use a high refresh rate to refresh and render the gaze area and differential area of ​​the display at the current moment, so as to avoid the phenomenon of the image of the gaze area at the previous moment still being high-definition content, causing a freeze.

[0009] In some embodiments, the above-mentioned method for determining the user's gaze area at the current moment based on pupil coordinates specifically includes: the electronic device determines a model based on the pupil coordinates and preset screen coordinates, and determines the screen coordinates corresponding to the pupil coordinates on the display; the electronic device determines the size of the gaze area, and determines the gaze area based on the screen coordinates and the size of the gaze area, and the screen coordinates are the center coordinates of the gaze area.

[0010] Based on this possible embodiment, the electronic device can accurately determine the gaze area at the current moment according to the pupil coordinates of the user and the size of the gaze area at the current moment.

[0011] In some embodiments, the method for determining the size of the gaze area specifically includes: the electronic device determines the ratio between the gaze area and the display area of ​​the display, and the ratio satisfies a preset formula, which is X+X×N+(1-2X)×M×N=K, wherein X represents the ratio between the gaze area and the display area of ​​the display, N represents the compression ratio of the image corresponding to the non-attention area, M represents the refresh ratio of each frame of the image corresponding to the non-attention area, and K represents the ratio between the actual resolution of the electronic device at the current moment and the rated resolution of the electronic device; the electronic device determines the size of the gaze area based on the product of the ratio and the display area of ​​the display.

[0012] Based on this possible embodiment, the electronic device can accurately determine the gaze area at the current moment according to a preset formula, which is simple and convenient.

[0013] In some embodiments, the method also includes: the electronic device determines a projection matrix of the target area based on position information of the target area; the target area is any one of a gaze area, a differential area, and a low-definition area, and the position information of the target area includes coordinate information and size information of the target area; the electronic device determines an image corresponding to the target area based on the projection matrix of the target area.

[0014] Based on this possible embodiment, the electronic device can determine the corresponding projection matrix according to the coordinate information and size information of the attention area and the non-attention area. Since the projection matrix can be used to determine the image corresponding to the area, the electronic device can accurately determine the corresponding image according to the projection matrix of each area, and then render the image corresponding to each area.

[0015] In some embodiments, the above-mentioned method of splicing the image corresponding to the rendered gaze area with the image corresponding to the rendered non-gaze area may specifically include: the electronic device splices the information of the image corresponding to the rendered gaze area and the information of the image corresponding to the rendered non-gaze area according to a preset format, wherein the image information includes the data of the image and the position of the image in the display area.

[0016] Based on this possible embodiment, in order to facilitate the transmission of images, the electronic device can splice the rendered image corresponding to the gaze area and the rendered image corresponding to the non-gaze area into a preset format. In this way, when the display receives the spliced ​​image, it can parse it to obtain the rendered complete image, and then display the rendered image.

[0017] On the other hand, an image processing device is provided, which is applied to an electronic device with a display, and the device includes: an acquisition unit, a processing unit and a stitching unit. The acquisition unit is configured to: acquire the pupil coordinates of the user at the current moment. The processing unit is configured to: determine the gaze area and non-gaze area corresponding to the display at the current moment according to the pupil coordinates of the user; render the image corresponding to the gaze area according to a first resolution, and render the image corresponding to the non-gaze area according to a second resolution, wherein the first resolution is greater than the second resolution. The stitching unit is configured to: stitch the rendered image corresponding to the gaze area with the rendered image corresponding to the non-gaze area, and display the stitched image on the display.

[0018] In some embodiments, the non-attention area includes a differential area and a low-definition area, the differential area is the area of ​​the attention area corresponding to the display at the first moment except the first area, the first area is the overlapping area of ​​the attention area corresponding to the display at the first moment and the attention area corresponding to the display at the current moment, and the first moment is the previous moment of the current moment; the low-definition area is the area of ​​the display except the attention area at the first moment and the attention area at the current moment. The processing unit is further configured to: render the image corresponding to the attention area at the current moment and the image corresponding to the differential area according to the first refresh rate, and render the image corresponding to the low-definition area according to the second refresh rate, and the first refresh rate is greater than the second refresh rate.

[0019] In some embodiments, the processing unit is specifically configured to: determine a model based on pupil coordinates and preset screen coordinates, and determine the screen coordinates corresponding to the pupil coordinates on the display; determine the size of the gaze area, and determine the gaze area based on the screen coordinates and the size of the gaze area, and the screen coordinates are the center coordinates of the gaze area.

[0020] In some embodiments, the processing unit is specifically configured to: determine a ratio between the gaze area and the display area of ​​the display, the ratio satisfies a preset formula, the preset formula is X+X×N+(1-2X)×M×N=K, wherein X represents the ratio between the gaze area and the display area of ​​the display, N represents the compression ratio of the image corresponding to the non-attention area, M represents the refresh ratio of each frame image corresponding to the non-attention area, and K represents the ratio between the actual resolution of the electronic device at the current moment and the rated resolution of the electronic device; the electronic device determines the size of the gaze area based on the product of the ratio and the display area of ​​the display.

[0021] In some embodiments, the processing unit is further configured to: determine the projection matrix of the target area based on the position information of the area; the target area is any one of the gaze area, the differential area, and the low-definition area, and the position information of the target area includes the coordinate information and size information of the target area; determine the image corresponding to the target area based on the projection matrix of the target area.

[0022] In some embodiments, the stitching unit is specifically configured to stitch the information of the rendered image corresponding to the gaze area and the information of the rendered image corresponding to the non-gaze area according to a preset format, wherein the image information includes the data of the image and the position of the image in the display area.

[0023] In another aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed on a computer, the computer executes the image processing method as described in any one of the above embodiments.

[0024] In another aspect, a computer program is provided. When the computer program is executed on a computer, the computer program enables the computer to execute the method for constructing an ETL system as described in any of the above embodiments.

[0025] On the other hand, a chip is provided, comprising a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run a computer program or instruction to implement the image processing method as described in any of the above embodiments.

[0026] Specifically, the chip provided in the present application also includes a memory for storing computer programs or instructions.

[0027] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium, wherein the computer-readable storage medium may be packaged together with the processor of the device, or may be packaged separately from the processor of the device, which is not limited in this application.

[0028] In this application, the names of the above devices do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the present disclosure, the following briefly introduces the drawings required to be used in some embodiments of the present disclosure. Obviously, the drawings described below are only drawings of some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can also be obtained based on these drawings. In addition, the drawings described below can be regarded as schematic diagrams, and are not limitations on the actual size of the product involved in the embodiments of the present disclosure, the actual process of the method, the actual timing of the signal, etc.

[0030] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application;

[0033] Figure 4 A schematic diagram of a gaze area and a non-gaze area of ​​a display provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of different refresh rates of another electronic device provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of a method for refreshing a low-definition area of ​​an electronic device provided in an embodiment of the present application;

[0036] Figure 7 A schematic diagram of a gaze area of ​​a display provided in an embodiment of the present application;

[0037] Figure 8 A schematic diagram of dividing a display into differential areas provided in an embodiment of the present application;

[0038] Fig. 9 A schematic diagram of dividing a low-definition area of ​​a display provided in an embodiment of the present application;

[0039] Fig.10 A flowchart of a data processing method provided in an embodiment of the present application;

[0040] Fig.11 A schematic diagram of image stitching provided in an embodiment of the present application;

[0041] Fig.12 A flowchart of another data processing method provided in an embodiment of the present application;

[0042] Fig.13A schematic diagram of another image processing device provided in an embodiment of the present application;

[0043] Fig.14 A schematic diagram of another image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in some embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments provided by the present disclosure, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present disclosure.

[0045] Unless the context requires otherwise, throughout the specification and claims, the term "comprise" and other forms thereof, such as the third person singular form "comprises" and the present participle form "comprising", are to be interpreted as open, inclusive, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "example", "specific example" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner.

[0046] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0047] When describing some embodiments, the expressions "coupled" and "connected" and their derivatives may be used. For example, when describing some embodiments, the term "connected" may be used to indicate that two or more components are in direct physical or electrical contact with each other. For another example, when describing some embodiments, the term "coupled" may be used to indicate that two or more components are in direct physical or electrical contact. However, the term "coupled" or "communicatively coupled" may also refer to two or more components that are not in direct contact with each other, but still cooperate or interact with each other. The embodiments disclosed herein are not necessarily limited to the contents of this document.

[0048] “At least one of A, B, and C” has the same meaning as “at least one of A, B, or C” and both include the following combinations of A, B, and C: A only, B only, C only, the combination of A and B, the combination of A and C, the combination of B and C, and the combination of A, B, and C.

[0049] “A and / or B” includes the following three combinations: A only, B only, and a combination of A and B.

[0050] As used herein, the term "if" is optionally interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined that" or "if [a stated condition or event] is detected" are optionally interpreted to mean "upon determining that" or "in response to determining that" or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0051] The use of "adapted to" or "configured to" herein is meant to be open and inclusive language that does not exclude devices adapted or configured to perform additional tasks or steps.

[0052] Additionally, the use of “based on” is meant to be open and inclusive, as a process, step, calculation, or other action “based on” one or more stated conditions or values ​​may, in practice, be based on additional conditions or values ​​beyond those stated.

[0053] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of variation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0054] The following explains the terms involved in the embodiments of the present application to facilitate the reader's understanding.

[0055] Refresh rate: Refresh rate refers to how many times the image displayed by the display device is redrawn per second, that is, the number of times the electronic device is refreshed per second, measured in Hertz (Hz). The higher the refresh rate, the more stable the image, the more natural and clear the image display, and the less impact on the user's eyes. The lower the refresh rate, the more severe the image flickers and jitters, and the faster the user's eyes get tired. Generally speaking, if the refresh rate can reach above 80Hz, the image flicker and jitter can be completely eliminated, and the user's eyes will not get tired too easily.

[0056] Resolution: It can be called resolution or resolution. Resolution refers to the number of dots displayed by the monitor in the horizontal and vertical directions. For example: 1024×768, 1280×1024, etc. The "1024" in 1024×768 refers to the number of dots in the horizontal direction of the screen, and "768" refers to the number of dots in the vertical direction of the screen. Resolution determines the fineness of the image. The higher the resolution of the image, the more pixels the image contains, and the clearer the image.

[0057] Normally, in order to meet the user's demand for high-definition content of images, it is necessary to use a higher resolution to render each frame of the image. At the same time, in order to make the image more stable, the electronic device also needs to use a higher refresh rate to render each frame of the image. However, the transmission capacity of the bandwidth of the electronic device and the rendering capacity of the GPU / graphics card are limited. If the image is processed at the maximum resolution supported by the electronic device, the frame rate that matches the refresh rate of the electronic device is often not achieved, and the processing pressure of the GPU / graphics card is increased. Therefore, it is necessary to reduce the processing pressure of the GPU as much as possible while meeting the user's demand for high-definition content.

[0058] In addition, the higher the rendering capability of a GPU / graphics card, the higher its price. In order not to increase costs, it is necessary to use the rendering capability of the GPU / graphics card reasonably. For example, in addition to ordinary 2D displays, multi-viewpoint light field displays based on light field technology need to process and transmit several to dozens of images at the same time. Therefore, how to use the rendering capability of the GPU / graphics card reasonably has become a demand.

[0059] After research and analysis, it was found that people are more sensitive to the clarity of the area they are looking at, and are relatively insensitive to areas outside the area they are looking at. Based on this principle, electronic devices can only perform high-definition rendering on the area they are looking at, thereby saving overall GPU / graphics card rendering overhead and transmission bandwidth.

[0060] In view of this, an embodiment of the present application provides an image processing method, which is applied to an electronic device with a display. The method may include: the electronic device obtains the pupil coordinates of the user, and determines the gaze area and non-gaze area of ​​the user on the display according to the pupil coordinates. The electronic device uses a high resolution to render the image of the gaze area, and uses a low resolution to render the image of the non-gaze area, that is, the electronic device can use different resolutions to render the same frame of image. Compared with using a high resolution to render each frame of image, the rendering pressure of the GPU of the electronic device is reduced. In addition, the electronic device can also splice the rendered image of the gaze area and the rendered image of the non-gaze area, and display the spliced ​​image on the display. Since the image of the gaze area in the spliced ​​image is a high-definition image, the user's demand for high-definition images is met.

[0061] It should be noted that in the embodiments of the present application, the electronic device may be a device with a display such as a television or a computer. The display of the electronic device may support refresh in different regions, and the refresh rate of each region may be different. For example, the display area of ​​the display of the electronic device may be divided into multiple sub-regions, and each sub-region may be displayed at a different refresh rate.

[0062] In one possible design, the electronic device may have multiple control components, each corresponding to a sub-area. Each control component has a driving circuit between each display pixel of the corresponding sub-area in the display area. In this way, each control component can control the output of the corresponding sub-area. That is, each control component can control the refresh rate and resolution of the corresponding sub-area.

[0063] The implementation of the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

[0064] like Figure 1 As shown, Figure 1 The present invention provides a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device may include a display 110 and a processor 120. The display 110 and the processor 120 may be communicatively connected. For example, the display 110 and the processor 120 may be connected via a system bus 130. Alternatively, the processor 120 may be embedded in the display 110.

[0065] The display 110 may be referred to as a display screen, a screen, etc. The display 110 may be used to display images. For example, the display may be a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, etc.

[0066] The processor 120 may include one or more GPUs and one or more central processing units (CPUs). The GPU may be used to render images. The CPU may be used to parse images and perform operations of the electronic device in the embodiments of the present application. For example, images may be spliced, a user's gaze area may be determined, and the like.

[0067] In one possible implementation, Figure 2 As shown, the electronic device may be provided with one or more cameras 210. The one or more cameras 210 may be provided below the display of the electronic device. Of course, the one or more cameras 210 may also be provided elsewhere in the electronic device, such as above the electronic device, or on the side. Not limited. Alternatively, in order to more accurately capture the user's facial image from multiple angles. A camera 210 may be provided on each edge of the display of the electronic device. The one or more cameras 210 may be communicatively connected to the processor 120.

[0068] The one or more cameras 210 may be used to capture images of the user and transmit the images of the user to the processor so that the processor may recognize the images of the user to obtain the pupil coordinates of the user.

[0069] For example, the one or more cameras 210 may include one or more red green blue (RGB) cameras and / or one or more infrared radiation (IR) cameras (only one RGB camera and three IR cameras are shown in the figure). Both the RGB camera and the IR camera may be used to collect image information of the user. The image information of the user may include a facial image and a pupil image of the user.

[0070] In one example, the electronic device can collect the user's facial image through an RGB camera, and collect the user's pupil image through an IR camera. Since the RGB refresh rate is higher but the resolution is lower, while the IR camera has a higher resolution. Therefore, the electronic device can perform facial recognition on the image taken by the RGB camera, calibrate the eye area and determine the coordinates of the eye area. Then, the electronic device can calibrate the eye area on the IR high-definition pupil image based on the coordinates of the eye area, and perform pupil detection in the eye area to calculate the user's pupil coordinates. That is, the electronic device only needs to identify the pupil in the pupil image taken by the IR camera, and there is no need to identify the user's eye area again, thereby reducing the amount of calculation of the processor.

[0071] Furthermore, in order to reduce the pressure on the IR camera, the electronic device can control one or more IR cameras to cyclically and alternately collect the user's pupil images.

[0072] In one example, an electronic device may be provided with a timing control module. The electronic device may output control information through the timing control module, and the control information may be used to control the working frequency of one or more IR cameras. For example, the control information may include the working time of each of the one or more IR cameras within a working cycle. For example, taking 3 IR cameras and the total duration of a working cycle of 3 hours as an example, the control information may include IR1-1, IR2-2, and IR3-3. Among them, "IR1", "IR2", and "IR3" are the identifiers of the IR cameras, "IR1-1" means that the IR1 camera works within the first hour, "IR2-2" means that the IR2 camera works within the second hour, and "IR3-3" means that the IR3 camera works within the third hour. In this way, the three IR cameras can be cyclically and alternately collected to ensure that the electronic device can continuously output pupil coordinates for equal durations.

[0073] It should be noted that when the electronic device is provided with multiple IR cameras, each IR camera may have a corresponding identifier, and the data of the pupil image captured by each IR camera may carry the identifier of the IR camera. The electronic device may determine the corresponding IR camera according to the identifier of the IR camera.

[0074] In another example, the electronic device can also control the one or more IR cameras to synchronously capture the user's pupil image. In this way, the electronic device can simultaneously capture multiple frames of the user's pupil image and determine the pupil coordinates corresponding to each frame of the pupil image. Furthermore, the electronic device can accurately determine the user's pupil coordinates based on the multiple pupil coordinates.

[0075] It should be noted that the pupil coordinates may be two-dimensional coordinates or three-dimensional coordinates. Specifically, it may be related to the settings of the IR camera and RGB camera of the electronic device. For example, when the IR camera and RGB camera of the electronic device support shooting three-dimensional images, the IR camera and RGB camera can shoot three-dimensional facial images, and the pupil coordinates of the user determined by the electronic device may be three-dimensional coordinates. When the IR camera and RGB camera of the electronic device do not support shooting three-dimensional images, the IR camera and RGB camera can shoot two-dimensional facial images, and the pupil coordinates of the user determined by the electronic device may be two-dimensional coordinates.

[0076] Furthermore, since the positions of the IR camera and the RGB camera of the electronic device are fixed, after the electronic device recognizes the face image taken by the RGB camera to obtain the coordinates of the human eye region and recognizes the pupil image taken by the IR camera to obtain the coordinates of the pupil, the coordinates of the human eye region can be converted into pupil coordinates based on the relative distance between the IR camera and the RGB camera.

[0077] In another example, the electronic device may be provided with an eye tracking module. The eye tracking module may track the movement of the human eye in real time. The eye tracking module may determine the pupil coordinates of the user based on the eye tracking algorithm. In this way, the electronic device may determine the pupil coordinates more accurately without the need for face recognition through multiple images.

[0078] In another example, the electronic device may also be provided with an infrared detection module. The infrared detection module may be used to detect whether a user exists. If the electronic device detects that no user is viewing the display during the use of the electronic device, or the electronic device cannot detect the pupil coordinates of the user, the electronic device may refresh each frame of the image using a low resolution and a low refresh rate. When the electronic device uses an infrared detection module to detect that the user is within a preset range or the distance from the electronic device is less than a preset distance, the electronic device may control the IR camera and the RGB camera to collect the user's pupil coordinates, thereby determining the user's gaze area, and rendering the image corresponding to the gaze area using a high resolution and a high refresh rate.

[0079] In another example, the electronic device may also be provided with a graphics card status monitoring module. The graphics card status module may be used to monitor the temperature and resource usage of the graphics card. When the temperature of the graphics card exceeds a preset temperature and / or the resource usage exceeds a preset threshold, the electronic device may prompt the user to turn off the display, or refresh each frame of the image using a low resolution and a low refresh rate. In this way, the security of the graphics card can be improved.

[0080] In one possible implementation, Figure 3 As shown, the electronic device provided in the embodiment of the present application may also include a human eye gaze point recognition module, a gaze area rendering module, a differential area rendering module, a low-definition area rendering module, and an image stitching module.

[0081] The eye gaze point recognition module can be used to determine the user's gaze area and non-gaze area in the display 110 according to the user's pupil coordinates. The specific method of determining the gaze area and non-gaze area can refer to the following description and will not be repeated here.

[0082] The gaze point rendering module can be used to render the image corresponding to the gaze area. The difference area rendering module is used to render the image corresponding to the difference area. The low-definition area rendering module is used to render the image corresponding to the low-definition area. The image stitching module can be used to stitch the images corresponding to each area to obtain a stitched image. The specific stitching method can be referred to the description below, which will not be repeated here.

[0083] Furthermore, in order to render the image more accurately, the non-attention area can be divided into a differential area (also referred to as a difference area) and a low-definition area. That is, the image can include an image corresponding to the attention area, an image corresponding to the differential area, and an image corresponding to the low-definition area. The method for determining the image corresponding to each area can refer to the following description and will not be repeated here.

[0084] The gaze area may refer to the area that the user mainly watches at a certain moment, or may refer to the area that the user's pupil is focused on at a certain moment. The differential area may refer to the area in the gaze area of ​​the previous moment of the current moment, excluding the overlapping area. The overlapping area may refer to the overlapping area between the gaze area of ​​the current moment and the gaze area of ​​the previous moment. The low-definition area may refer to the other areas in the display area of ​​the display except the gaze areas of two consecutive moments.

[0085] For example, Figure 4 In a and b, area A (the area formed by the solid line) may refer to the gaze area at the current moment, and area B may refer to the gaze area at the first moment (the area formed by the dotted line). The first moment is the moment before the current moment. That is, the user's gaze area moves from area B to area A. Area C (that is, the area in area B except the overlapping area of ​​area A and area B) is the differential area. Figure 4 In a and b, the area other than area A and area B in the display area of ​​the electronic device can be a low-definition area.

[0086] For example, Figure 4 As shown in c, when there is no overlapping area between area A and area B, area B can be both the fixation area at the previous moment and the difference area. Figure 4 The area other than area A and area B in c may be a low-definition area.

[0087] After introducing the concepts of fixation area, difference area and low-definition area, the following describes the methods for determining fixation area, difference area and low-definition area respectively:

[0088] 1. Method for determining the gaze area.

[0089] In a possible implementation, the electronic device can determine the model according to the pupil coordinates and the preset screen coordinates, and determine the screen coordinates corresponding to the pupil coordinates on the display. Then, the electronic device determines the gaze area of ​​the display at the current moment according to the size of the gaze area and the screen coordinates.

[0090] Among them, the preset screen coordinate determination model can be used to determine the screen coordinates corresponding to the pupil coordinates on the display. The preset screen coordinate determination model can be pre-configured by the electronic device, or can be obtained by training the electronic device according to a preset algorithm, multiple pupil coordinates and corresponding screen coordinates. The input of the preset screen coordinates is the pupil coordinates of the user, and the output is the screen coordinates. The screen coordinates can also be called the gaze point coordinates.

[0091] In one example, when the electronic device is initially started, the electronic device can display multiple calibration points on the display and prompt the user to look at each calibration point in turn. For example, when the electronic device is first started, before displaying the content, the electronic device can display calibration points in the upper left corner, lower right corner, lower left corner, and upper right corner of the display in turn, and output prompt information to prompt the user to look at the calibration point. At the same time, the electronic device can collect pupil images when the user looks at each calibration point in turn, and determine the corresponding pupil coordinates. In this way, the electronic device can obtain the coordinates of multiple calibration points and the corresponding pupil coordinates. Furthermore, the electronic device can use the multiple calibration points and the corresponding pupil coordinates as sample data, use a preset algorithm to train these sample data, and obtain a preset screen coordinate determination model. For example, the preset algorithm can be a neural network algorithm, such as a deep convolutional neural network algorithm.

[0092] Furthermore, in order to verify the accuracy of the preset screen coordinate determination model, the electronic device may continue to collect multiple calibration points and the corresponding pupil coordinates of the user. The electronic device may use the coordinates of the multiple calibration points and the corresponding pupil coordinates as test samples to test the preset screen coordinate determination model. If the error between the screen coordinates output by the preset screen coordinate determination model and the coordinates of the calibration points is within a preset range, it means that the accuracy of the preset screen coordinate determination model is high. If the error exceeds the preset range, the preset screen coordinate determination model may continue to be trained.

[0093] Of course, the electronic device can also fit the multiple calibration points and the corresponding pupil coordinates to obtain a linear relationship between the calibration points and the pupil coordinates. Based on the linear relationship, the electronic device can determine the screen coordinates (calibration point coordinates) corresponding to the user's pupil coordinates.

[0094] In order to verify the accuracy of the linear relationship, the electronic device can randomly display multiple calibration points on the display and obtain the pupil coordinates corresponding to each calibration point. Then, the electronic device can verify the accuracy of the linear relationship based on the coordinates of the multiple calibration points and the corresponding pupil coordinates. If the difference between the coordinates of the calibration point determined by the linear relationship and the calibration point corresponding to the pupil coordinates is within a preset range, it means that the accuracy of the linear relationship is high. If the difference exceeds the preset range, it means that the error of the linear relationship is high, and the electronic device can refit according to the coordinates of the multiple calibration points and the corresponding pupil coordinates to obtain a new linear relationship.

[0095] It should be noted that, in the embodiment of the present application, the electronic device may determine the ratio between the gaze area and the display area of ​​the display, and then the electronic device may determine the size of the gaze area according to the product of the ratio and the display area.

[0096] In one example, the ratio between the gaze area and the display area of ​​the display satisfies a preset formula. For example, the preset formula may be the following formula 1.

[0097] X+X×N+(1-2X)×M×N=K Formula 1

[0098] Where X represents the ratio between the fixation area and the display area of ​​the display. N represents the compression ratio of the image corresponding to the non-fixation area. M represents the refresh ratio of each frame of the image corresponding to the non-fixation area. K represents the ratio between the actual resolution of the display and the rated resolution.

[0099] Specifically, the image compression ratio N may refer to the ratio between the number of pixels actually displayed in each sub-area of ​​the display and the number of pixels that can be displayed. For example, if the image compression ratio is 1 / 4, it means that the number of pixels actually displayed in each sub-area is 1 / 4 of the number of pixels that can be displayed in the sub-area. For example, if the number of pixels that can be displayed in each sub-area is 240, the number of pixels actually displayed in each sub-area after compression is 60.

[0100] For example, the display of the electronic device may include 32×9=288 sub-areas, which are arranged in sequence. Each sub-area has a corresponding identifier (such as a number). For example, the identifiers of the 288 sub-areas are 1 to 288. The total bandwidth of the electronic device for transmitting images is 7680×

[0101] 2160@120Hz. "7680×2160" indicates the rated resolution of an electronic device, and 120Hz is the refresh rate of the electronic device. The number of pixels that can be displayed in each sub-area of ​​the display is 7680 / 32=240.

[0102] For example, Figure 5As shown in a, when the refresh rate of the electronic device is 120 Hz, the electronic device can refresh the image once every two refresh cycles. Figure 5 As shown in b, when the refresh rate of the electronic device is 240 Hz, the electronic device can refresh the image once in each refresh cycle.

[0103] It should be noted that Figure 5 In a, the 240Hz of the GPU / graphics card indicates that the refresh rate of the display is 240Hz, and the 120Hz of the corresponding image of the display indicates that the refresh rate of the display refreshes the image is 120Hz. That is, the electronic device can refresh a frame of image within two refresh cycles. Figure 5 In b, the 240 Hz of the GPU / graphics card indicates that the refresh frequency of the display is 240 Hz, and the 240 Hz of the corresponding image of the display indicates that the refresh rate of the display refreshes the image is 240 Hz. That is, the electronic device can refresh one frame of image in each refresh cycle.

[0104] By the above Figure 5 As can be seen from the description of a and b, in the embodiment of the present application, by increasing the refresh rate of the electronic device, the rate at which the display refreshes the image can be increased. Compared with the electronic device refreshing the image using a low refresh rate, the delay of the display can be reduced.

[0105] The refresh ratio M of each frame of the image corresponding to the non-attention area may refer to the ratio of the area of ​​each refresh of the image corresponding to the non-attention area to the total area. For example, if the image corresponding to the non-attention area is refreshed by 1 / 3 at a certain moment, the refresh ratio of the image corresponding to the non-attention area at that moment is 1 / 3. In this way, the image of the entire non-attention area can be completely updated for 3 consecutive times.

[0106] The refresh ratio of the image corresponding to the non-attention area is 1 / 3, which means that the electronic device refreshes 1 / 3 of the image corresponding to the non-attention area each time. Figure 6 As shown, the electronic device refreshes the image of area a in the non-attention area at time 1, refreshes the image of area b in the non-attention area at time 2, and refreshes the image of area c in the non-attention area at the third time. Among them, time 1, time 2, and time 3 are three consecutive moments. Time 1 is the previous moment of time 2, and time 2 is the previous moment of time 3. The sizes of area a, area b, and area c can be the same or different, but the sum of the areas of the three areas is the entire non-attention area.

[0107] It should be noted that, for the image corresponding to the non-attention area, each refresh of the non-attention area only refreshes 1 / 3 of the image, thereby reducing the amount of GPU rendering.

[0108] Furthermore, after determining the ratio between the gaze area and the display area of ​​the display, the electronic device may determine the size of the display area according to the size of the display area of ​​the display and the ratio, and use the screen coordinates corresponding to the pupil coordinates as the center coordinates of the display area.

[0109] In one example, in order to make the refresh rate of the electronic device 240Hz, the capacity of the image can be reduced by half, that is, the actual resolution of the display is set to 3840×2160, that is, K=1 / 2. Taking the refresh ratio of the non-attention area as 1 / 3 and the compression ratio of the image corresponding to the non-attention area as 1 / 4 as an example, combined with the above formula one, it can be calculated that the ratio between the attention area and the display area of ​​the display is 5 / 13. Combined with the above display including 288 sub-areas, the size of the attention area is 288×5 / 13, which is approximately equal to 110.7. Theoretically, the attention area can include 10×10=100 sub-areas, but the display has a total of 9 partitions in the vertical direction, so the attention area can be set to 12×9=108 sub-areas. In this way, the electronic device can expand the screen coordinates corresponding to the pupil coordinates as the center and 108 sub-areas as the size to obtain the attention area. For example, Figure 7 As shown in a in FIG, point d is the screen coordinate corresponding to the pupil coordinate, and the fixation area is area D1. Area D1 includes 108 sub-areas.

[0110] It should be noted that in the embodiment of the present application, the shape of the gaze area is a rectangle. When the electronic device is expanded with the screen coordinates corresponding to the pupil coordinates as the center, if the expanded area exceeds the display, the boundary of the display is used as the boundary of the gaze area. Figure 7 As shown in b, when the gaze area calculated by the electronic device includes 108 sub-areas, point d is the screen coordinate corresponding to the pupil coordinate, and area D2 is the gaze area calculated by the electronic device, but the left boundary of area D2 exceeds the display area of ​​the display. Therefore, the electronic device can use area D3 (including 90 sub-areas) as the gaze area.

[0111] In another possible implementation, in order to reduce the pressure on the processor of the electronic device, the size of the gaze area of ​​the electronic device can be preset. For example, when the display includes 32×9 sub-areas, the gaze area can include 9×9=81 sub-areas. Figure 7 After determining the screen coordinate d corresponding to the pupil coordinate of the user, the electronic device can take the screen coordinate d as the center, and the size of the gaze area is the sum of the sizes of the 81 sub-areas as the gaze area. In this way, the electronic device only needs to determine the pupil coordinates of the electronic device at each moment, and then directly determine the user's gaze area according to the preset gaze area size.

[0112] 2. Method for determining the differential area.

[0113] In a possible implementation, the electronic device may perform a Boolean operation on the gaze areas at two consecutive moments to calculate a differential area.

[0114] In one example, the electronic device may be provided with a memory (or a buffer), and the electronic device may store information of the gaze area at multiple consecutive moments. For example, the coordinates of the gaze area may be stored. After determining the gaze area of ​​the user at the current moment, the electronic device may compare the coordinates of the gaze area at the current moment with the coordinates of the gaze area of ​​the user at the previous moment stored to determine the differential area.

[0115] For example, if the coordinates of the lower left corner of the current fixation area are located on the border of the fixation area at the previous moment, the difference area can be Figure 4 The C region in a.

[0116] For another example, if the coordinates of the lower left corner of the current fixation area are within the fixation area at the previous moment, the difference area can be Figure 4 The C area in b.

[0117] For another example, if the coordinates of the lower left corner of the current fixation area are outside the fixation area at the previous moment, the difference area can be Figure 4 The B area.

[0118] 3. Method for determining low-definition areas.

[0119] In a possible implementation, after determining the gaze area and the differential area, the electronic device may determine the low-definition area according to the difference between the display area and the gaze area and the differential area.

[0120] For example, the electronic device can determine the identifiers of the multiple sub-areas included in the low-definition area based on the identifiers of all sub-areas of the stored display area, the identifiers of the multiple sub-areas included in the gaze area, and the identifiers of the multiple sub-areas included in the differential area. Furthermore, the electronic device can determine the low-definition area of ​​the display based on the identifiers of the multiple sub-areas included in the low-definition area.

[0121] In another possible implementation, in order to reduce the amount of calculation, the electronic device may use the entire display area as a low-definition area.

[0122] It should be noted that in the embodiment of the present application, when the electronic device uses high resolution and high refresh rate and low resolution and low refresh rate to render the image of the same area, the rendered image has high resolution and high refresh rate. That is, a high-definition image can cover a low-definition image, or a high-definition image can be superimposed on a low-definition image, and the superimposed image is a high-definition image. Therefore, the electronic device renders the entire display area as a low-definition area, and can still make the gaze area a high-definition image, which does not affect the user experience, thereby meeting the user's demand for high-definition content.

[0123] After describing the method for determining the gaze area, the difference area, and the low-definition area provided in the embodiment of the present application, the method for determining the image corresponding to the gaze area, the image corresponding to the difference area, and the image corresponding to the low-definition area provided in the embodiment of the present application will be described below.

[0124] In a possible embodiment, the electronic device may determine the projection matrix corresponding to each area according to the position information of each area, and determine the image corresponding to the area to be rendered according to the projection matrix.

[0125] The projection matrix can be used to determine the image corresponding to the area that needs to be rendered. The rendering area can include the gaze area, the difference area and the low-definition area.

[0126] For example, the electronic device may have a virtual camera. For example, the virtual camera may be a software program of the electronic device. The virtual camera may be a set of data parameters of the electronic device. The data parameters may be used to identify the position, orientation, viewing angle, etc. of the rendered image. The size of the viewport of the virtual camera may be adjusted according to the projection matrix. For example, when the projection matrix is ​​a projection matrix corresponding to the gaze area, the electronic device may adjust the viewport of the virtual camera according to the projection matrix so that the adjusted viewport of the virtual camera can capture the image corresponding to the gaze area (the shape of the image captured by the viewport of the virtual camera is a rectangle). The electronic device may determine the image corresponding to the gaze area according to the viewport of the virtual camera, and render the image to obtain the rendered image of the gaze area.

[0127] In one example, the projection matrix (camera.projection Matrix) of the gaze area can be obtained according to the preset projection matrix of the virtual camera, and one or more matrices. The preset projection matrix can be preset for the electronic device, for example, each component in the preset projection matrix is ​​related to the data parameters of the virtual camera. The one or more matrices can be determined according to the position information of the rendering area. The order of the one or more matrices is consistent with the order of the preset projection matrix. For example, the one or more matrices and the preset projection matrix can both be 4×4 matrices.

[0128] For example, the preset projection matrix of an electronic device

[0129]

[0130] Among them, fovy can identify the vertical angle of the virtual camera's frustum, aspect represents the aspect ratio of the virtual camera, f represents the far plane distance of the virtual camera, and n represents the near plane distance of the virtual camera. These parameters of the virtual camera can be set as needed, for example, fov=60, n=0.1, f=1000, aspect=1.775, without restriction.

[0131] The position information of the rendering area r=Rect(x, y, w, h). Wherein, (x, y) may be the coordinates of the rendering area (for example, the lower left corner coordinates, the lower right corner coordinates, etc.), w may be the width of the rendering area (or the sum of the widths of the multiple sub-areas included in the rendering area), and h may be the height of the rendering area (or the sum of the heights of the multiple sub-areas included in the rendering area). Then the one or more matrices may be the following m2 and m3.

[0132] For example, taking the matrix as a 4-order matrix,

[0133] Then the projection matrix corresponding to the rendering area is M=m3×m2×m1.

[0134] Furthermore, in order to simplify the amount of calculation for determining the image corresponding to the rendering area, the parameters in the position information of the rendering area may be normalized. That is, x, y, w, and h in m2 and m3 are normalized. The normalization process may refer to the prior art and will not be described in detail.

[0135] It should be noted that the above matrix is ​​illustrated with a 4th-order matrix as an example, and the projection matrix of the virtual camera can also be a matrix of other orders, for example, a 3rd-order matrix, without limitation. Accordingly, the order of the matrix constructed by the position information of the rendering area is consistent with the order of the projection matrix.

[0136] In one example, when the rendering area is a gaze area, the electronic device can determine the image corresponding to the gaze area based on a matrix constructed by the lower left corner coordinates, height, and width of the gaze area and a preset projection matrix, and then the electronic device can render the image corresponding to the gaze area to obtain a rendered image of the gaze area.

[0137] In another example, when the rendering area is a difference area, and the difference area is Figure 4 A or Figure 4When the differential area shown in c (area C or area A), since the shape of the differential area is a rectangle, the electronic device can calculate the projection matrix corresponding to the differential area according to the product of the matrix constructed by the lower left corner coordinates, height, and width of the differential area and the preset projection matrix, and determine the image corresponding to the differential area according to the projection matrix corresponding to the differential area. The electronic device can render the image corresponding to the differential area to obtain the rendered image of the differential area.

[0138] In another example, when the rendering area is a difference area, and the difference area is Figure 4 When the differential area (area C) shown in b is used, since the shape of the differential area is an irregular image, the electronic device can divide the differential area into multiple rectangles.

[0139] For example, Figure 8 As shown in a and b, the electronic device can divide the difference area (area C) into area C1 and area C2. In this way, the electronic device can determine the projection matrices corresponding to area C1 and area C2 respectively, and determine the images corresponding to each area according to the projection matrices corresponding to area C1 and area C2. Then, the electronic device can render the image corresponding to area C1 and the image corresponding to area C2 respectively to obtain the rendered image corresponding to the difference area.

[0140] For example, Figure 8 As shown in c, the electronic device can divide the differential area (area C) into area C1, area C2, and area C3. In this way, the electronic device can determine the projection matrices corresponding to area C1, area C2, and area C3, respectively, and determine the images corresponding to each area according to the projection matrices corresponding to area C1, area C2, and area C3. Then, the electronic device can render the image corresponding to area C1, the image corresponding to area C2, and the image corresponding to area C3, respectively, to obtain the rendered image corresponding to the differential area.

[0141] In another example, when the rendering area is a low-definition area, since the low-definition area is an irregular shape, the electronic device can divide the low-definition area into a plurality of rectangles. Figure 4 a, such as Fig. 9 As shown, the electronic device can divide the low-definition area into a plurality of rectangular areas (such as area D1 to area D4). In this way, the electronic device can calculate the projection matrix corresponding to each matrix area, and further determine the image corresponding to the low-definition area.

[0142] Furthermore, in order to reduce the amount of calculation of the electronic device, the electronic device can use the entire display area as a low-definition area and render the low-definition area using a low resolution and a low refresh rate. For example, in combination with the above electronic device, the low-definition area corresponding to moment 1 can be used as a complete rendering area. In this way, the electronic device can calculate the projection matrix of the low-definition area corresponding to moment 1, and then determine the image corresponding to the low-definition area, and perform low-definition rendering on the image corresponding to the low-definition area to obtain a rendered image.

[0143] After describing the above method for determining the fixation area and the non-fixation area (including the differential area and the low-definition area) and the method for determining the image corresponding to each area, the following is a method for determining the fixation area and the non-fixation area (including the differential area and the low-definition area) according to the present invention. Figure 1 The electronic device shown illustrates the image processing method provided in the embodiment of the present application.

[0144] like Fig.10 As shown, an image processing method provided by an embodiment of the present application can be applied to Figure 1 The electronic device shown may also be applied to a processor in an electronic device. The method may include S1001 to S1003.

[0145] S1001. Obtain the pupil coordinates of the user at the current moment, and determine the gaze area and non-gaze area of ​​the display at the current moment according to the pupil coordinates of the user.

[0146] Specifically, this step may refer to the above-mentioned method for determining the gaze area, which will not be described in detail.

[0147] S1002: Render an image corresponding to a focus area according to a first resolution, and render an image corresponding to a non-focus area according to a second resolution.

[0148] The first resolution is greater than the second resolution. The first resolution and the second resolution can be set as needed, for example, the first resolution can be 7680×2160, and the second resolution can be 3840×1080, without limitation.

[0149] In a possible implementation, the electronic device may be pre-set with multiple different resolutions. In this way, the electronic device may select a first resolution from multiple different resolutions to render the image corresponding to the gaze area and select a second resolution to render the image of the non-gaze area according to the pre-set instructions, so that the rendered image corresponding to the gaze area and the rendered image corresponding to the non-gaze area can be obtained.

[0150] S1003: splice the rendered image corresponding to the gaze area and the rendered image corresponding to the non-gaze area, and display the spliced ​​image on a display.

[0151] The electronic device can splice the rendered image corresponding to the gaze area and the rendered image corresponding to the non-gaze area according to a preset format to obtain a rendered image, and then the electronic device can display the spliced ​​image on a display. Since the gaze area of ​​the spliced ​​image is high-definition content, the needs of users are met. In addition, the non-gaze area of ​​the spliced ​​image is low-definition content, which reduces the rendering amount of the GPU.

[0152] In one possible implementation, the electronic device can encapsulate the data and position information of the image corresponding to the rendered gaze area and the data and position information of the image corresponding to the rendered non-gaze area into data in a preset format, and transmit the data to the display, so that the display parses the encapsulated data to obtain the image corresponding to each sub-area.

[0153] The image location information is used to indicate the location of the image in the corresponding display area and the scaling ratio (also referred to as the compression ratio). The image location information may include identifiers of multiple sub-areas of the display area corresponding to the image. The scaling ratio of the image may refer to the ratio between the compressed size of the image and the actual size. In this way, the electronic device can reduce the amount of data transmission in the transmission bandwidth by compressing the image.

[0154] In one example, the position information of the image corresponding to each rendered area can be as follows: Fig.11 As shown in a. Fig.11 In a, the current fixation area (area A) does not overlap with the first fixation area (area B), that is, the first fixation area is also a differential area. The low-definition area may be one third of the entire display area, or may be an area in the area that does not overlap with the fixation area.

[0155] For example, the electronic device may stitch the image corresponding to the current fixation area and the image corresponding to the non-fixation area into a Fig.11 The format shown in b. Fig.11 In b, each sub-region is respectively provided with image information. For example, the first region may be provided with image information corresponding to the fixation region, the second region may be provided with image information corresponding to the difference region, and the third region may be provided with image information corresponding to the low-definition region.

[0156] It should be noted that for Fig.11 The image of the fixation area is a high-resolution and high-refresh-rate image. Therefore, the image information of a sub-area in the first area may include Fig.11 Data of an image of a sub-region of the fixation region in a and an identification of the sub-region.

[0157] Since the image in the differential area is a low-resolution, high-refresh-rate image, the electronic device can Fig.10 The information of the compressed image in the difference area of ​​a is set in the second area.

[0158] The image information of a sub-region in the second region may include information of images corresponding to multiple sub-regions of the differential region. For example, the number of sub-regions (a total of 108 sub-regions) and the position of the image of the differential region in the display region are as follows: Fig.11 The electronic device compresses the image of the differential region at a compression ratio of 1 / 4, and the compressed image of the differential region is Fig.11 The number of corresponding sub-regions in b is 27. That is, Fig.11 The information of the image of a sub-region of the second region b may include Fig.11 The image data of the four sub-regions of the difference region in a and the identifiers of the four sub-regions.

[0159] Since the image in the low-definition area is a low-resolution and high-refresh-rate image, the electronic device can compress the image in the low-definition area and set it in the third area.

[0160] The image information of a sub-region in the third region may include information of images corresponding to multiple sub-regions in the low-definition region. For example, the number of sub-regions (a total of 24 sub-regions) and the position of the image in the low-definition region in the display region are as follows: Fig.11 The electronic device compresses the image in the low-definition area according to a compression ratio of 1 / 4, and the image in the differential area after compression is Fig.11 The number of corresponding sub-regions in b is 6. That is, Fig.11 The image information of a sub-region of the third region in b may include Fig.11 The image data of the four sub-regions of the low-definition area in a and the identifiers of the four sub-regions.

[0161] It should be noted that, from the description of the method for determining the gaze area, the number of sub-areas of the gaze area at the current moment calculated by the electronic device is 110.7. For the convenience of calculation, the number of sub-areas of the gaze area is adjusted to 108. Therefore, Fig.11 The blank area in b can be used to set the image data of the remaining sub-areas.

[0162] In another possible implementation, of course, in order to facilitate splicing, the electronic device can Fig.11 The images corresponding to each area in a are as follows Fig.11 The format shown in c is used for splicing. Fig.11 The image in c may also have an information header. The information header may be used to indicate Fig.11 Each sub-region in c corresponds to the sub-region when displayed on the display and the scaling ratio when displayed. For example, Fig.11 In c, the information header of the first area may include the number of the sub-area corresponding to a sub-area in the first area when displayed on the display and the scaling ratio is 1. The information header of the second area may include the numbers of multiple sub-areas corresponding to a sub-area in the second area and a sub-area in the third area when displayed on the display and the scaling ratio is 1 / 4.

[0163] Furthermore, the processor of the electronic device can transmit the Fig.11 b or Fig.11 The information of the image shown in c is sent to the display through the transmission bandwidth. After receiving the image information from the processor, the display can parse and obtain the image data corresponding to the gaze area and the image data corresponding to the non-gaze area, and display the image in the corresponding area.

[0164] Based on the technical solution provided by this embodiment, after obtaining the pupil coordinates of the user, the electronic device can determine the user's gaze area and non-gaze area on the display according to the pupil coordinates. Furthermore, the electronic device can use high resolution to render the image of the gaze area, and use low resolution to render the image of the non-gaze area, that is, the electronic device can use different resolutions to render the same frame of image. Compared with using high resolution to render the complete image, the rendering pressure of the GPU of the electronic device is reduced. In addition, the electronic device can also splice the rendered image of the gaze area and the rendered image of the non-gaze area, and display the spliced ​​image on the display. Since the image of the gaze area in the spliced ​​image is a high-definition image, the user's demand for high-definition images is met.

[0165] In a possible embodiment, Fig.12 As shown, the method provided in the embodiment of the present application may also include:

[0166] S1201: Render an image corresponding to the gaze area and an image corresponding to the difference area at the current moment according to a first refresh rate, and render a low-definition area according to a second refresh rate.

[0167] The first refresh rate is greater than the second refresh rate. The difference area and the low-definition area can refer to the above Figure 4 The description is not repeated here.

[0168] Based on this possible embodiment, the electronic device uses a high refresh rate to render the image of the gaze area and the image of the differential area at the current moment. Since the image displayed in the differential area was a high-definition image at the previous moment of the current moment, the image of the differential area can be rendered using a high refresh rate and a low resolution. While reducing the rendering pressure of the GPU, the image of the differential area and the image of the gaze area can be refreshed using a high refresh rate to avoid the image of the differential area and the image of the gaze area being out of sync, resulting in image freezes. At the same time, the electronic device can use a low refresh rate to render the image of the low-definition area, further reducing the processing pressure of the GPU.

[0169] It should be pointed out that the various embodiments of the present application can learn from or refer to each other. For example, the same or similar steps, method embodiments, system embodiments and device embodiments can all refer to each other without limitation.

[0170] The embodiment of the present application can divide the image processing device into functional modules or functional units according to the above method example. For example, each functional module or functional unit can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules or functional units. Among them, the division of modules or units in the embodiment of the present application is schematic, which is only a logical function division. There may be other division methods in actual implementation.

[0171] like Fig.13 , which is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device is applied to an electronic device with a display. The device includes: an acquisition unit 1301, a processing unit 1302 and a splicing unit 1303.

[0172] The acquisition unit 1301 is configured to: acquire the pupil coordinates of the user at the current moment. For example, the acquisition unit 1301 can be used to perform Fig.10 S1001 in the figure. The processing unit 1302 is configured to: determine the gaze area and the non-gaze area corresponding to the display at the current moment according to the pupil coordinates of the user; render the image corresponding to the gaze area according to the first resolution, and render the image corresponding to the non-gaze area according to the second resolution, wherein the first resolution is greater than the second resolution. For example, the processing unit 1302 can be used to perform Fig.10 S1002 and Fig.12 The stitching unit 1303 is configured to stitch the rendered image corresponding to the gaze area with the rendered image corresponding to the non-gaze area, and display the stitched image on the display. For example, the stitching unit 1303 can be used to perform Fig.10 Medium S1003.

[0173] In some embodiments, the non-attention area includes a differential area and a low-definition area, the differential area is the area of ​​the attention area corresponding to the display at the first moment except the first area, the first area is the overlapping area of ​​the attention area corresponding to the display at the first moment and the attention area corresponding to the display at the current moment, and the first moment is the previous moment of the current moment; the low-definition area is the area of ​​the display except the attention area at the first moment and the attention area at the current moment. The processing unit 1302 is also configured to: render the image corresponding to the attention area at the current moment and the image corresponding to the differential area according to the first refresh rate, and render the image corresponding to the low-definition area according to the second refresh rate, and the first refresh rate is greater than the second refresh rate.

[0174] In some embodiments, the processing unit 1302 is specifically configured to: determine a model based on pupil coordinates and preset screen coordinates, and determine the screen coordinates corresponding to the pupil coordinates on the display; determine the size of the gaze area, and determine the gaze area based on the screen coordinates and the size of the gaze area, and the screen coordinates are the center coordinates of the gaze area.

[0175] In some embodiments, the processing unit 1302 is specifically configured to: determine a ratio between the gaze area and the display area of ​​the display, the ratio satisfies a preset formula, the preset formula is X+X×N+(1-2X)×M×N=K, wherein X represents the ratio between the gaze area and the display area of ​​the display, N represents the compression ratio of the image corresponding to the non-gaze area, M represents the refresh ratio of each frame image corresponding to the non-gaze area, and K represents the ratio between the actual resolution of the electronic device at the current moment and the rated resolution of the electronic device; the electronic device determines the size of the gaze area based on the product of the ratio and the display area of ​​the display.

[0176] In some embodiments, the processing unit 1302 is further configured to: determine the projection matrix of the target area based on the position information of the area; the target area is any one of the gaze area, the differential area, and the low-definition area, and the position information of the target area includes the coordinate information and size information of the target area; determine the image corresponding to the target area based on the projection matrix of the target area.

[0177] In some embodiments, the stitching unit 1303 is specifically configured to stitch the information of the rendered image corresponding to the gaze area and the information of the rendered image corresponding to the non-gaze area according to a preset format, wherein the image information includes the data of the image and the position of the image in the display area.

[0178] When implemented by hardware, the acquisition unit 1301 in the embodiment of the present application can be integrated on the communication interface, and the processing unit 1302 and the splicing unit 1303 can be integrated on the processor. Fig.14 shown.

[0179] Fig.14 A schematic diagram of the structure of another possible communication device of the image processing device involved in the above embodiment is shown. The communication device includes: a processor 1402 and a communication interface 1403. The processor 1402 is used to control and manage the actions of the device, for example, to execute the steps performed by the above processing unit 1302 and the splicing unit 1303, and / or to execute other processes of the technology described herein. The communication interface 1403 is used to support the communication between the device and other network entities, for example, to execute the steps performed by the above acquisition unit 1301. The device may also include a memory 1401 and a bus 1404, and the memory 1401 is used to store program codes and data of the device.

[0180] Among them, the memory 1401 can be a memory in the device, etc., and the memory can include a volatile memory, such as a random access memory; the memory can also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory can also include a combination of the above types of memory.

[0181] The processor 1402 may be a processor that implements or executes various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present application. The processor may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The processor may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0182] The bus 1404 may be an Extended Industry Standard Architecture (EISA) bus, etc. The bus 1404 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.14 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0183] Fig.14 The device in the embodiment may also be a chip. The chip includes one or more than two (including two) processors 1402 and a communication interface 1403.

[0184] Optionally, the chip further includes a memory 1401, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor 1402. A portion of the memory 1401 may also include a non-volatile random access memory (NVRAM).

[0185] In some implementations, the memory 1401 stores the following elements, execution modules or data structures, or a subset thereof, or an extended set thereof.

[0186] In the embodiment of the present application, the corresponding operation is performed by calling the operation instruction stored in the memory 1401 (the operation instruction may be stored in the operating system).

[0187] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0188] Some embodiments of the present disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium), which stores computer program instructions. When the computer program instructions are executed on a computer, the computer executes the method for constructing an ETL system as described in any of the above embodiments.

[0189] Exemplarily, the above-mentioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks or magnetic tapes, etc.), optical disks (e.g., CDs (Compact Disks), DVDs (Digital Versatile Disks), etc.), smart cards and flash memory devices (e.g., EPROMs (Erasable Programmable Read-Only Memory), cards, sticks or key drives, etc.). The various computer-readable storage media described in the present disclosure may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0190] Some embodiments of the present disclosure also provide a computer program product, for example, the computer program product is stored on a non-transitory computer-readable storage medium. The computer program product includes computer program instructions, and when the computer program instructions are executed on a computer, the computer program instructions cause the computer to execute the method described in the above embodiments.

[0191] Some embodiments of the present disclosure further provide a computer program. When the computer program is executed on a computer, the computer program enables the computer to execute the method described in the above embodiments.

[0192] The beneficial effects of the above-mentioned computer-readable storage medium, computer program product and computer program are the same as the beneficial effects of the methods described in some of the above-mentioned embodiments, and will not be repeated here.

[0193] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0194] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0196] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be thought of by any person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device has a display, and the method comprises: Obtaining pupil coordinates of the user at the current moment, and determining a fixation area and a non-fixation area corresponding to the display at the current moment according to the pupil coordinates; wherein the size of the fixation area is the product of a first ratio and a display area of ​​the display, and the first ratio satisfies a preset formula, and the preset formula is: X+X×N+(1-2X)×M×N=K; Wherein, X represents the first ratio, N represents the compression ratio of the image corresponding to the non-attention area, M represents the refresh ratio of each frame of the image of the non-attention area, and K represents the ratio between the actual resolution of the electronic device and the rated resolution of the electronic device at the current moment; Rendering the image corresponding to the gaze area according to a first resolution, and rendering the image corresponding to the non-gaze area according to a second resolution, wherein the first resolution is greater than the second resolution; The rendered image corresponding to the gaze area is spliced ​​with the rendered image corresponding to the non-gaze area, and the spliced ​​image is displayed on the display.

2. The method according to claim 1, characterized in that: The non-attention area includes a differential area and a low-definition area, the differential area is an area in the attention area corresponding to the display at the first moment except the first area, the first area is an overlapping area between the attention area of ​​the display at the first moment and the attention area of ​​the display at the current moment, and the low-definition area is an area in the display except the attention area at the first moment and the attention area at the current moment; The first moment is a moment before the current moment; the method further includes: The image corresponding to the gaze area at the current moment and the image corresponding to the differential area are rendered according to a first refresh rate, and the image corresponding to the low-definition area is rendered according to a second refresh rate, wherein the first refresh rate is greater than the second refresh rate.

3. The method according to claim 1 or 2, characterized in that: The determining the gaze area of ​​the user at the current moment according to the pupil coordinates includes: Determine the screen coordinates corresponding to the pupil coordinates on the display according to the pupil coordinates and the preset screen coordinate determination model; The gaze area is determined according to the screen coordinates and the size of the gaze area, wherein the screen coordinates are the center coordinates of the gaze area.

4. The method according to claim 2, characterized in that: The method also includes: determining a projection matrix corresponding to the target area according to position information of the target area, the target area being any one of the gaze area, the differential area, and the low-definition area, and the position information including coordinate information and size information of the target area; and determining an image corresponding to the target area according to the projection matrix.

5. The method according to any one of claims 1, 2 and 4, characterized in that: The step of splicing the rendered image corresponding to the gaze area with the rendered image corresponding to the non-gaze area comprises: The information of the rendered image corresponding to the fixation area and the information of the rendered image corresponding to the non-fixation area are spliced ​​according to a preset format, wherein the image information includes the data of the image and the position of the image in the display area.

6. A data processing device, characterized in that: Applied to an electronic device, the electronic device has a display, and the device includes an acquisition unit, a processing unit and a splicing unit: The acquisition unit is configured to: acquire the pupil coordinates of the user at the current moment; The processing unit is configured to: determine, according to the pupil coordinates, a fixation area and a non-fixation area corresponding to the display at the current moment; wherein the size of the fixation area is the product of a first ratio and a display area of ​​the display, and the first ratio satisfies a preset formula, and the preset formula is: X+X×N+(1-2X)×M×N=K; Wherein, X represents the first ratio, N represents the compression ratio of the image corresponding to the non-attention area, M represents the refresh ratio of each frame of the image of the non-attention area, and K represents the ratio between the actual resolution of the electronic device and the rated resolution of the electronic device at the current moment; The processing unit is further configured to: render the image corresponding to the gaze area according to a first resolution, and render the image corresponding to the non-gaze area according to a second resolution, wherein the first resolution is greater than the second resolution; The splicing unit is configured to splice the rendered image corresponding to the gaze area with the rendered image corresponding to the non-gaze area, and display the spliced ​​image on the display.

7. The device according to claim 6, characterized in that The non-attention area includes a differential area and a low-definition area, the differential area is an area other than the first area in the attention area corresponding to the display at the first moment, the first area is an overlapping area between the attention area of ​​the display at the first moment and the attention area of ​​the display at the current moment, and the low-definition area is an area of ​​the display other than the attention area at the first moment and the attention area at the current moment; the first moment is a moment before the current moment; The processing device is further configured to: render the image corresponding to the gaze area at the current moment and the image corresponding to the differential area according to a first refresh rate, and render the image corresponding to the low-definition area according to a second refresh rate, wherein the first refresh rate is greater than the second refresh rate.

8. The device according to claim 6 or 7, characterized in that The processing device is specifically configured to: determine a model according to the pupil coordinates and preset screen coordinates, and determine the screen coordinates corresponding to the pupil coordinates on the display; determine the gaze area according to the screen coordinates and the size of the gaze area, and the screen coordinates are the center coordinates of the gaze area.

9. The device according to claim 7, characterized in that The processing unit is further configured to: determine a projection matrix corresponding to the target area according to position information of the target area, wherein the target area is any one of the gaze area, the difference area, and the low-definition area, and the position information includes coordinate information and size information of the target area; An image corresponding to the target area is determined according to the projection matrix.

10. The device according to any one of claims 6, 7 and 9, characterized in that The splicing unit is configured to splice the information of the rendered image corresponding to the gaze area and the information of the rendered image corresponding to the non-gaze area according to a preset format, wherein the image information includes the data of the image and the position of the image in the display area.

11. A processing device, characterized in that: include: A processor and a communication interface; the communication interface is coupled to the processor, and the processor is used to run a computer program or instruction to implement the image processing method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions. When a computer executes the instructions, the computer executes the image processing method according to any one of claims 1 to 5.

Images