Image processing method and device

Abstract

The present application provides an image processing method, the method comprising: obtaining an eye image of a target user; determining a first gaze position of the target user on a display interface according to the eye image; determining a first area corresponding to the first gaze position in the image to be displayed, and a second area other than the first area in the image to be displayed according to the first gaze position; rendering the first area and the second area respectively to obtain a first rendered image and a second rendered image, merging the first rendered image and the second rendered image to obtain a first target image, wherein the image quality of the first rendered image is higher than that of the second rendered image. The technical method provided in the present application determines an attention point according to an eye image, thereby rendering images near the attention point with high quality and images outside the attention point with low quality, thereby reducing hardware performance requirements and pressure while ensuring the user's viewing experience, and does not require complex eye tracking equipment, and is relatively low in cost.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus, computer device and computer-readable storage medium. Background Art

[0002] With the advancement of technology, the quality of videos has been continuously improved. The current mainstream video resolution has reached the 4K level (for example, 4096×2160), and even some devices and video sources already support 8K resolution (for example, 7680x4320). Although the improvement of video quality can greatly improve the user's viewing experience, as the high resolution continues to increase, the demand for device hardware capabilities is also increasing. This is because: First, from traditional 1080p to 4K and 8K, each time the image quality is improved, the amount of pixels required to be rendered is about 4 times that of the previous time. Secondly, the real-time requirements for high-resolution videos are not low, and for videos with more details, if the fluency is not enough, it will also affect the user's viewing experience. Summary of the invention

[0003] The purpose of this application is to provide an image processing method, apparatus, computer device and computer-readable storage medium to solve the following technical problem: it is difficult to balance image quality, required processing performance and user viewing experience when rendering high-resolution images.

[0004] One aspect of an embodiment of the present application provides an image processing method, the method comprising:

[0005] Obtain the current frame of the target user's eye image through the camera;

[0006] Acquire an eye image of a target user, wherein the eye image is a continuous multi-frame image;

[0007] Determine a first gaze position of the target user on the display interface according to the eye image;

[0008] Determine, according to the first gaze position, a first area in the image to be displayed corresponding to the first gaze position, and a second area in the image to be displayed except the first area;

[0009] The first area and the second area in the image to be displayed are rendered respectively to obtain a first rendered image and a second rendered image, and the first rendered image and the second rendered image are merged to obtain a first target image, wherein the image quality of the first rendered image is higher than that of the second rendered image.

[0010] Optionally, acquiring the eye image of the target user includes:

[0011] Acquire a facial image of the target user;

[0012] Using at least two different thresholds to perform binarization processing on the face image to obtain at least two binarized face images;

[0013] The eye image of the target user is determined based on at least two of the binary face images.

[0014] Optionally, determining the eye image of the target user according to at least two of the binary face images includes:

[0015] Determine the pupil area of ​​the human eye according to the circular areas in at least two of the binary face images;

[0016] The eye image is determined according to the pupil area of ​​the human eye.

[0017] Optionally, determining a first gaze position of the target user on the display interface according to the eye image includes:

[0018] Processing the eye image of the current frame using a pre-trained gaze position prediction model to obtain the pupil movement offset of the target user's eye in the current frame;

[0019] Determine the initial predicted gaze position of the target user in the current frame according to the initial predicted gaze position of the target user in the previous frame and the pupil movement offset of the target user in the current frame;

[0020] The first gaze position of the target user in the current frame is determined according to the initial predicted gaze position of the target user in the current frame and the predetermined eye states of the target user when viewing a plurality of preset calibration points on the display interface.

[0021] Optionally, determining the first gaze position of the target user in the current frame according to the initial predicted gaze position of the target user in the current frame and a predetermined eye state of the target user when viewing a plurality of preset calibration points on the display interface includes:

[0022] Determining the eye state of the target user in a current frame according to the eye image of the target user in a current frame;

[0023] Determine at least three preset calibration points adjacent to the initial predicted gaze position according to the eye state of the target user in a current frame and the predetermined eye states of the target user when viewing a plurality of preset calibration points on the display interface;

[0024] The first gaze position is determined according to the initial predicted gaze position and at least three of the preset calibration points adjacent to the initial predicted gaze position.

[0025] Optionally, determining the first gaze position according to the initial predicted gaze position and at least three preset calibration points adjacent to the initial predicted gaze position includes:

[0026] The initial predicted gaze position of the target user in the current frame is adjusted, and when the sum of the distances between the adjusted initial predicted gaze position and the at least three preset calibration points reaches a minimum, the adjusted initial predicted gaze position is used as the first gaze position.

[0027] Optionally, before acquiring the eye image of the target user, the method further includes:

[0028] Acquire an eyeball image of the target user when the preset calibration point on the display interface is the gaze position;

[0029] The eye state of the target user when viewing the preset calibration point on the display interface is determined according to the eye image of the target user when the preset calibration point on the display interface is the gaze position.

[0030] Optionally, the determining a first gaze position of the target user on the display interface according to the eye image further includes:

[0031] An initial predicted gaze position of the target user in the first frame is determined according to the predetermined eye states of the target user when viewing a plurality of preset calibration points on the display interface and the eye image of the target user in the first frame.

[0032] Optionally, determining, according to the first gaze position, a first area in the image to be displayed corresponding to the first gaze position, and a second area in the image to be displayed except the first area, comprises:

[0033] The image to be displayed is divided into a plurality of rectangular areas, and an area including the first gaze position is used as the first area, and an area excluding the first gaze position is used as the second area.

[0034] Optionally, determining, according to the first gaze position, a first area in the image to be displayed corresponding to the first gaze position, and a second area in the image to be displayed except the first area, comprises:

[0035] In the image to be displayed, a circular area with a certain radius and the first gaze position as the center is used as the first area, and an area outside the first area is used as the second area.

[0036] Optionally, the rendering the first area and the second area in the image to be displayed respectively to obtain a first rendered image and a second rendered image includes:

[0037] Rendering the first region in the image to be displayed with original image quality to obtain the first rendered image;

[0038] Down-sampling the second region in the image to be displayed or the image to be displayed, and rendering the down-sampled second region in the image to be displayed or the image to be displayed to obtain the second rendered image.

[0039] Optionally, the merging the first rendered image and the second rendered image to obtain a first target image includes:

[0040] Performing feathering processing on the edge of the first rendered image;

[0041] The first rendered image that has undergone feathering processing is superimposed with the second rendered image to obtain the first target image.

[0042] Optionally, the method further comprises:

[0043] Determine a motion parameter of the first gaze position according to a plurality of consecutive first gaze positions of the target user on the display interface before a current moment;

[0044] Determine a second gaze position of the target user at a current moment according to the motion parameter of the first gaze position and a plurality of consecutive first gaze positions before a current moment;

[0045] When the first gaze position of the target user at the current moment is not determined, determining, according to the second gaze position, a third area in the image to be displayed corresponding to the second gaze position, and a fourth area in the image to be displayed except the third area;

[0046] The third area and the fourth area in the image to be displayed are rendered respectively to obtain a third rendered image and a fourth rendered image, and the third rendered image and the fourth rendered image are merged to obtain a second target image, wherein the image quality of the third rendered image is higher than that of the fourth rendered image.

[0047] Optionally, the eye image is acquired by a monocular camera.

[0048] One aspect of an embodiment of the present application further provides an image processing device, including:

[0049] An eye image acquisition module is used to acquire an eye image of a target user, wherein the eye image is a continuous multi-frame image;

[0050] A first gaze position determining module, configured to determine a first gaze position of the target user on the display interface according to the eye image;

[0051] an area determination module, configured to determine, according to the first gaze position, a first area in the image to be displayed corresponding to the first gaze position, and a second area in the image to be displayed except the first area;

[0052] A rendering module is used to render the first area and the second area in the image to be displayed respectively to obtain a first rendered image and a second rendered image, and merge the first rendered image and the second rendered image to obtain a first target image, wherein the image quality of the first rendered image is higher than that of the second rendered image.

[0053] One aspect of an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned image processing method when executing the computer program.

[0054] One aspect of an embodiment of the present application provides a computer-readable storage medium, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned image processing method when executing the computer program.

[0055] The image processing method, apparatus, computer device, and computer-readable storage medium provided in the embodiments of the present application have the following advantages:

[0056] By obtaining the target user's eye image and determining the target user's gaze position on the display interface based on the eye image, the area corresponding to the gaze position in the displayed image is rendered with high quality, and the image in the remaining area is rendered with low quality. This reduces hardware performance requirements and pressure while ensuring the user's viewing experience, and does not require complex eye tracking equipment, so the cost is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram showing a simulation of human eye observation is shown;

[0058] Figure 2 A schematic diagram schematically shows an application environment of the image processing method of the present application;

[0059] Figure 3 A flowchart of an image processing method according to Embodiment 1 of the present application is schematically shown;

[0060] Figure 4 for Figure 3Flow chart of sub-steps of step S300;

[0061] Figure 5 Schematically shows Figure 4 An example of performing multi-threshold binarization processing on the face image in step S402;

[0062] Figure 6 for Figure 3 Flow chart of sub-steps of step S302;

[0063] Figure 7 An example of eye tracking data is schematically shown;

[0064] Figure 8 for Figure 6 Flow chart of the sub-steps of step S604;

[0065] Fig. 9 The preset calibration points of the display interface are schematically shown;

[0066] Fig.10 A flowchart schematically shows the newly added steps of the image processing method according to the first embodiment of the present application;

[0067] Fig.11 Figure 3 Flow chart of newly added sub-steps of step S302;

[0068] Fig.12 for Figure 3 Flow chart of sub-steps of step S306;

[0069] Fig.13 for Fig.12 Flow chart of sub-steps of step S1206;

[0070] Fig.14 A flowchart schematically shows the newly added steps of the image processing method according to the first embodiment of the present application;

[0071] Fig.15 A block diagram schematically shows an image processing device according to a second embodiment of the present application;

[0072] Fig.16 The schematic diagram shows a hardware architecture of a computer device suitable for implementing the image processing method according to the third embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0074] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0075] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0076] Figure 1 A simulation diagram of human eye observation is schematically shown.

[0077] Although the human eye has a wide field of vision, about 220 degrees horizontally and about 135 degrees vertically, only about 6% of the cones are used to obtain clear visual details, and the rods, which account for the vast majority, can only provide blurred visual information. And the rods can only provide grayscale color information. Figure 1 It can be seen that the image information that the human eye can really recognize well is only a very small area near the point of attention of the human eye (the dotted area in the figure), which only occupies a very small part of the picture and is reflected in the viewing angle of only 1 to 2 degrees.

[0078] In view of this, for high-resolution videos, if the video can be appropriately downsampled and the parts with low precision requirements (areas outside the human eye's attention point) are rendered at low resolution, the pressure on machine performance can be alleviated to a great extent, achieving the goals of improving video smoothness and extending hardware life without reducing the user's viewing experience.

[0079] At present, the preferred method for determining the point of human eye attention is to use an eye tracker, which has high accuracy. However, the disadvantage of this method is that it requires professional eye tracking equipment, which is very inconvenient and costly. Based on the above description, this application proposes a method for realizing human eye tracking based on eye images obtained by a camera and then performing image processing based on the point of human eye attention, which can reduce the requirements of image processing on machine performance, relieve the pressure on machine performance, and does not require complex and expensive professional equipment.

[0080] Figure 2 The following is a schematic diagram schematically showing the application environment of the image processing method of the present application.

[0081] like Figure 2 As shown, the computer device 20 includes a display screen 21 and a camera 22, as well as other devices such as a processor (not shown). The computer device 20 can play a video on the display screen 21, and a user 23 can watch the video played on the display screen 21.

[0082] The computer device 20 can be various computers or mobile terminals, such as notebooks, desktops, tablet computers, smart phones, etc.

[0083] In this application, in order to alleviate the performance pressure on the computer device 20 caused by playing high-resolution videos, the computer device 20 will perform local high-precision rendering of the played video according to the gaze position (or attention point) of the user 23 on the display screen 21. Specifically, the area corresponding to the user's gaze position in the video image to be displayed is rendered with high precision, for example, it is rendered with the original high resolution, and the area outside the area corresponding to the user's gaze position is downsampled and rendered with a reduced resolution. In this way, the image of the user's gaze position area is still rendered with high precision to ensure the user's viewing effect, and the area outside the user's gaze position has a lower impact on the user's viewing effect, so it is rendered with a reduced resolution, which can alleviate the performance pressure of the rendering video on the computer device 20.

[0084] In the present application, the computer device 20 can collect the face image of the user 23 through the camera 22, and then extract the eye image from the face image, and then determine the gaze position of the user on the display screen 21 according to the eye image of the user, so as to perform local high-precision rendering of the played video based on the gaze position. In this way, the camera of the present application of the computer device can realize eye tracking, without complex and expensive professional equipment, with low cost and wide application range.

[0085] A number of embodiments will be provided below, and each of the embodiments provided below can be used to implement the image processing solution described above. For ease of understanding, the following exemplary description will be made with a computer device / server as the execution subject.

[0086] Embodiment 1

[0087] This embodiment provides an image processing method, which is applied to an electronic device / computer device equipped with a camera. The specific technical details and effects can be found below.

[0088] Figure 3 The flowchart of the image processing method according to the first embodiment of the present application is schematically shown.

[0089] like Figure 3 As shown, the image processing method in the first embodiment of the present application may include:

[0090] Step S300, obtaining an eye image of a target user, wherein the eye image is a continuous multi-frame image;

[0091] Step S302, determining a first gaze position of the target user on the display interface according to the eye image;

[0092] Step S304, determining, according to the first gaze position, a first area in the image to be displayed corresponding to the first gaze position, and a second area in the image to be displayed except the first area;

[0093] Step S306: render the first area and the second area in the image to be displayed respectively to obtain a first rendered image and a second rendered image, and merge the first rendered image and the second rendered image to obtain a first target image, wherein the image quality of the first rendered image is higher than that of the second rendered image.

[0094] In the above step S300, the target user is a user watching a video played by an electronic device / computer device, and the camera of the electronic device / computer device collects an image of the target user at a certain frame rate, and obtains an eye image of the target user from the collected image. Then, the eye state of the target user can be determined based on the eye image of the target user.

[0095] In this application, the eye image is a plurality of frames of eye images collected continuously. That is, in this application, the camera will continuously acquire images of the target user, and the eye image corresponding to the current moment is referred to as the eye image of the current frame in this article.

[0096] In the above step S302, after the eye image of the target user is acquired, the first gaze position of the target user on the display interface can be determined according to the eye image of the target user.

[0097] As an example, a pre-trained gaze position prediction model may be used to process the eye image of the target user to obtain the first gaze position of the target user on the display interface.

[0098] As another example, the movement angle of the target user's eyeball relative to a reference can be first obtained based on the target user's eye image, such as the reference being the state of the user's eyeball looking straight ahead, such as the reference being the state of the user's eyeball looking straight at the display screen of the electronic device. Then, the current spatial coordinates of the pupil are confirmed based on the movement angle of the eyeball. For example, by obtaining the movement angle of the user's eyeball relative to the reference, the rotation angle and direction of the pupil can be determined, with the center of the user's eyeball as the center (or origin), and the length and direction of the line connecting the pupil and the center are known, and the spatial coordinates of the user's pupil can be obtained in this space. Finally, the current spatial coordinates of the pupil are mapped to the two-dimensional coordinate system of the display interface, and the point where the pupil lands on the two-dimensional coordinate system is marked as the gaze position of the eyeball.

[0099] In the present application, the display interface refers to the interface on the display screen of a computer device / electronic device for displaying a video, which may be the entire area or a partial area of ​​the display screen.

[0100] In the above step S304, a first area corresponding to the first gaze position in the image to be displayed and a second area other than the first area in the image to be displayed are determined according to the first gaze position.

[0101] As an example, the image to be displayed is divided into a plurality of rectangular areas, and the area including the first gaze position is taken as the first area, and the area not including the first gaze position is taken as the second area. The area including the first gaze position means that the first gaze position is located within the rectangular area or on the border of the rectangular area. When the first gaze position is located on the border of the rectangular area, the rectangular area including the border where the first gaze position is located is regarded as the first area.

[0102] As another example, in the image to be displayed, a circular area with a certain radius of the center of the first gaze position is used as the first area, and an area outside the first area is used as the second area. That is, a circular area is determined as the first area with the first gaze position as the center, and the size of the circular area depends on the distance between the target user and the display interface. Specifically, the distance between the target user and the display interface can be determined according to the eye movement distance when the target user views different preset calibration points on the display interface. As described below, if the user's eye movement distance is large when viewing different calibration points (i.e., the eye needs to be moved in a large range), it means that the user is close to the display interface, so the visual range on the display interface is small, and a smaller radius can be selected (i.e., a smaller circular area is selected as the first area); on the contrary, if the user's eye movement distance is small when viewing different calibration points (i.e., the eye does not need to be moved in a large range), it means that the user is far away from the display interface, so the visual range on the display interface is large, and a larger radius needs to be selected (i.e., a larger circular area is selected as the first area).

[0103] In the above step S306, after determining the first area corresponding to the first gaze position in the image to be displayed and the second area other than the first area according to the first gaze position of the target user on the display interface, the first area and the second area in the image to be displayed are rendered respectively to obtain a first rendered image and a second rendered image, and the first rendered image and the second rendered image are merged to obtain a first target image, wherein the image quality of the first rendered image is higher than that of the second rendered image. In other words, the image to be displayed can be locally rendered with high precision according to the first gaze position, thereby reducing the image quality of other areas while ensuring the image quality of the area corresponding to the first gaze position, thereby alleviating performance pressure while ensuring the user viewing experience.

[0104] As an example, the image to be displayed in the current frame can be downsampled according to the first gaze position of the target user on the display interface. For example, the original resolution of the first area corresponding to the first gaze position in the image to be displayed is maintained, while the second area (other areas in the image to be displayed except the area corresponding to the first gaze position) is downsampled to reduce the resolution of the images in these areas, thereby reducing the performance required for rendering and alleviating the performance pressure of the device.

[0105] As another example, the image quality of the current frame to be displayed can be enhanced according to the first gaze position of the target user on the display interface. For example, the image quality of the first area corresponding to the first gaze position in the image to be displayed is enhanced (for example, upsampling is performed to increase the resolution of the first area), while the original image quality is maintained for the second area (other areas of the image to be displayed except the area corresponding to the first gaze position), that is, the image quality is enhanced only for some areas. This can reduce the performance required for rendering and alleviate the performance pressure of the device.

[0106] It should be understood that in the present application, the image quality of the first rendered image is higher than that of the second rendered image, which may be that the resolution of the first rendered image is higher than that of the second rendered image, for example, the first rendered image is 4K and the second rendered image is 1080P, or other image parameters of the second rendered image that affect the image quality or rendering pressure are lower than those of the first rendered image.

[0107] The image processing method of the embodiment of the present application has the following advantages:

[0108] By obtaining the target user's eye image and determining the target user's gaze position on the display interface based on the eye image, the area corresponding to the gaze position in the displayed image is rendered with high quality, and the image in the remaining area is rendered with low quality. This reduces hardware performance requirements and pressure while ensuring the user's viewing experience, and does not require complex eye tracking equipment, so the cost is low.

[0109] Some optional embodiments are provided below.

[0110] In an exemplary embodiment, a facial image of a target user may be captured by a camera, and then an eye image of the target user may be determined based on the facial image. Figure 4 As shown, step S300 may include: step S400, obtaining the facial image of the target user; step S402, binarizing the facial image using at least two different thresholds to obtain at least two binary facial images; step S404, determining the eye image of the target user based on at least two binary facial images.

[0111] In the above step S400, the target user may be imaged by a camera, and then a face image may be extracted from the imaged face. The face image may be extracted using a known face segmentation method or face detection method, or using a pre-trained face extraction model.

[0112] In the above step S402, after obtaining the face image of the target user, the face image is binarized using at least two different thresholds to obtain at least two binary face images. As an example, three different thresholds can be used to binarize the face image to obtain three binary images.

[0113] In the above step S404, the eye image of the target user is determined based on at least two of the binarized face images. Specifically, the eye image of the target user can be determined based on the binarized face images through the following steps. First, the pupil area of ​​the human eye is determined based on the circular area in at least two of the binarized face images, that is, a circular area is searched in each binarized face image, and then the position of the circular area in each binarized face image is compared. If the positions of these circular areas in the binarized face images are the same, the circular area can be considered as the pupil area of ​​the human eye. Then, the eye image is determined based on the pupil area of ​​the human eye. For example, in the face image of the target user, the pupil area of ​​the human eye can be appropriately enlarged outward to a certain extent, which can be regarded as the eye area, and then the image of this area is captured from the face image as the eye image.

[0114] The principle of the above method for determining the eye image is: since most human eyes (Asians) have black pupils, the human eyes are black under multiple binary thresholds, and the human pupils are all round, so you only need to search the circular areas of all binary images to find the human pupil area, and then you can find the human eye area by appropriately enlarging the area, and then capture the image of the area as the eye image.

[0115] The above method is described using Asians as an example, but for people from other regions, appropriate adjustments can be made or other methods can be used to determine the eye image from the face image, such as extracting the eye image from the face image using a pre-trained eye detection model.

[0116] Through the above scheme, the face image is collected by the camera of the electronic device / computer device, and then the eye image of the target user can be obtained by processing the face image. It is simple, convenient, and low-cost, and no additional equipment is required.

[0117] Figure 5 An example of multi-threshold binarization processing on a face image is shown.

[0118] like Figure 5As shown, the upper left picture in the figure is the collected face image of the target user. In this example, the face image is binarized using three different thresholds to obtain three binary face images. It can be seen from the three binary face images that only the pupil area of ​​the human eye is a black circular area. Therefore, the pupil area of ​​the human eye can be located by searching for the circular area in the three binary face images. After the pupil area of ​​the human eye is located, it is appropriately expanded to obtain the eye area, and then the eye image is obtained.

[0119] In an exemplary embodiment, the eye image may be processed using a pre-trained gaze position prediction model to obtain the first gaze position of the target user on the display interface, such as Figure 6 As shown, step S302 may include: step S600, using a pre-trained gaze position prediction model to process the eye image of the current frame to obtain the human eye pupil movement offset of the target user in the current frame; step S602, determining the initial predicted gaze position of the target user in the current frame according to the initial predicted gaze position of the target user in the previous frame and the human eye pupil movement offset of the target user in the current frame; step 604, determining the first gaze position of the target user in the current frame according to the initial predicted gaze position of the target user in the current frame, and the predetermined eye state of the target user when viewing multiple preset calibration points on the display interface.

[0120] In the above step S600, the eye image of the current frame is processed using a pre-trained gaze position prediction model to obtain the pupil movement offset of the target user in the current frame.

[0121] As an example, the pre-trained gaze position prediction model can use the Transformer network (proposed by Google), because the gaze position of the human eye is not only related to the current moment, but will continue to move based on the position of the previous moment. The pre-trained gaze position prediction model can predict the gaze position of the human eye in the current frame by taking the eye images of consecutive frames as input, that is, the eye images of the target user obtained in step S300 are continuously input into the pre-trained gaze position prediction model, and then the eye gaze position of the current frame is obtained based on the continuous input of multiple groups of eye images. It should be understood that for the pre-trained gaze position prediction model, the input is the eye image, and the output is the three-dimensional offset of the gaze position of the human eye. The reason why the offset is three-dimensional is that the head of the target user will move, resulting in the offset of the actual gaze position being a three-dimensional movement. The network cannot directly output the actual gaze position, because the different shooting distances of the camera and the different distances between the user's eyes and the screen will cause the user's eyeball movement amplitude to be different. Therefore, the gaze position prediction model can only output the movement offset of a human pupil.

[0122] The gaze position prediction model can be trained using common training methods and open source datasets. Figure 7 The training was performed using eye tracking data from the TEyeD[2] database.

[0123] In the above step S602, after obtaining the pupil motion offset of the target user in the current frame, the initial predicted gaze position of the target user in the current frame is obtained in combination with the initial predicted gaze position of the target user in the previous frame. That is, the initial predicted gaze position of the target user in the previous frame is adjusted according to the pupil motion offset to obtain the initial predicted gaze position of the target user in the current frame. The initial predicted gaze position is a predicted gaze position determined based on the through-hole motion offset output by the gaze position prediction model, which is a three-dimensional position, and the three-dimensional coordinate system can be established with the center of the display interface as the origin.

[0124] In the above step S604, after the initial predicted gaze position of the target user in the current frame is determined, the initial predicted gaze position needs to be transformed into the coordinate system of the display interface, so as to determine the first gaze position of the target user in the current frame on the display interface.

[0125] Specifically, the first gaze position of the target user in the current frame can be determined according to the predetermined eye states of the target user when viewing multiple preset calibration points on the display interface and the initial predicted gaze position of the target user in the current frame.

[0126] In an exemplary embodiment, if Figure 8 As shown, step S604 may include: step S800, determining the eye state of the target user in the current frame according to the eye image of the target user in the current frame; step S802, determining at least three preset calibration points adjacent to the initial predicted gaze position according to the eye state of the target user in the current frame and the eye states of the target user predetermined when viewing a plurality of preset calibration points on the display interface; step S804, determining the first gaze position according to the initial predicted gaze position and at least three preset calibration points adjacent to the initial predicted gaze position.

[0127] In the above step S800, the eye state of the target user in the current frame is determined according to the eye image of the target user in the current frame, and the eye state can be represented by the offset of the pupil relative to the center of the eye.

[0128] In the above step S802, the eye state of the target user in the current frame is compared with the eye states of multiple preset calibration points of the target user when viewing the display interface, and at least three eye states similar to the eye state of the target user in the current frame are selected from the eye states of the multiple preset calibration points. Then, the preset calibration points corresponding to the at least three eye states are used as the at least three preset calibration points adjacent to the initial predicted gaze position.

[0129] As an example, three eye states close to the eye state of the target user in the current frame are selected from the eye states of multiple preset calibration points, and then the preset calibration points corresponding to the three eye states are used as the three preset calibration points adjacent to the initial predicted gaze position.

[0130] In the above step S804, the first gaze position is determined according to the initial predicted gaze position and at least three preset calibration points adjacent to the initial predicted gaze position. That is, the initial predicted gaze position is placed on the spatial plane where the display interface is located according to at least three preset calibration points adjacent to the initial predicted gaze position, thereby obtaining the first gaze position.

[0131] Exemplarily, the initial predicted gaze position of the target user in the current frame is adjusted, and when the sum of the distances between the adjusted initial predicted gaze position and the at least three preset calibration points reaches a minimum, the adjusted initial predicted gaze position is used as the first gaze position. In other words, after the initial predicted gaze position is obtained, the first gaze position is obtained by calculating the distance between the position and at least three nearest preset calibration points and appropriately modifying the position so that the sum of the at least three distances is minimized.

[0132] As an example, the first gaze position P may be determined by the following formula:

[0133] min((dis(P-P1)-L1)+(dis(P-P2)-L2)+(dis(P-P3)-L3)+dis(P-P0))

[0134] Among them, P0 is the initial predicted gaze position of the target user in the current frame, which can be regarded as the predicted position output by the gaze position prediction model, and P1, P2, and P3 are three calibration points adjacent to the initial predicted gaze position P0 among multiple preset calibration points. L1, L2, and L3 are the distances between P0 and P1, P2, and P3. By solving the above formula / equation, the most approximate human eye gaze position can be obtained as the first gaze position.

[0135] See also Fig. 9 , Fig. 9The schematic diagram shows the preset calibration points of the display interface. In this application, 5 calibration points are preset on the display interface ( Fig. 9 The circled area is located at the center and four corners of the display interface.

[0136] In an exemplary embodiment, when the target user uses an electronic device configured with the image processing method of the present application, the preset calibration points are used for calibration in advance, so as to determine the distance of the target user relative to the display interface and the eye state of each preset calibration point, so as to implement the method steps related to the calibration points in this article, such as Fig.10 As shown, the method also includes: step S1000, obtaining an eye image of the target user when the preset calibration point on the display interface is the gaze position; step S1002, determining the eye state of the target user when viewing the preset calibration point on the display interface based on the eye image of the target user when the preset calibration point on the display interface is the gaze position.

[0137] In the above step S1000, the eye image of the target user when the preset calibration point on the display interface is the gaze position can be obtained by first collecting the facial image of the target user when he or she looks at each preset calibration point, and then using the above method to obtain the eye image of the target user when the preset calibration point on the display interface is the gaze position based on the facial image.

[0138] In the above step S1002, the eye state of the target user when viewing the preset calibration point on the display interface is determined based on the eye image of the target user when the target user takes the preset calibration point on the display interface as the gaze position. The eye state can be represented by the offset of the pupil relative to the center of the eye.

[0139] Furthermore, in an exemplary embodiment, in the above step S1002, the eye movement distance of the target user when viewing each preset calibration point can also be obtained. Fig. 9 The preset calibration points are shown as an example. When the target user looks at the calibration point in the center, the eye movement distance can be regarded as 0, and the eye movement distance when watching the other 4 calibration points is obtained based on this. If the user moves a large distance when watching different calibration points (i.e., the eye needs to be moved in a large range), it means that the user is close to the display interface, so the visual range on the display interface is small; conversely, if the user moves a small distance when watching different calibration points (i.e., the eye does not need to be moved in a large range), it means that the user is far away from the display interface, so the visual range on the display interface is large.

[0140] In an exemplary embodiment, in order to facilitate determining the first gaze position of the target user on the display interface, such as Fig.11As shown, step S302 also includes: step S1100, determining the initial predicted gaze position of the target user in the first frame according to the predetermined eye state of the target user when viewing multiple preset calibration points on the display interface and the eye image of the target user in the first frame.

[0141] That is, after obtaining the eye image of the target user in the first frame, since there is only one frame of eye image at this time, the pupil offset cannot be output through the preset gaze position prediction model. At this time, it is necessary to determine the user's initial predicted gaze position in the first frame based on the eye state of multiple preset calibration points and the eye state of the eye image in the first frame.

[0142] The initial predicted gaze position of the first frame can be determined in the following manner: first, three eye states that are close to the eye state of the first frame are found from the eye states of multiple preset calibration points, and then the initial predicted gaze position of the first frame is determined according to the three preset calibration points corresponding to the three eye states. For example, a point with the smallest sum of distances from the three preset calibration points is determined as the initial predicted gaze position of the first frame.

[0143] After the initial predicted gaze position of the first frame is determined, the corresponding initial predicted gaze position of each subsequent eye image frame can be determined by adding the initial predicted gaze position of the previous frame to the pupil movement offset of the current frame.

[0144] In an exemplary embodiment, if Fig.12 As shown, step S306 may include: step S1200, rendering the first area in the image to be displayed with the original image quality to obtain a first rendered image; step S1202, down-sampling the second area in the image to be displayed or the image to be displayed, and rendering the down-sampled second area in the image to be displayed or the image to be displayed to obtain a second rendered image.

[0145] In the above step S1200, the first region in the image to be displayed is rendered with the original image quality to obtain a first rendered image. For example, the image components in the first region are rendered with the original resolution of the image to be displayed to obtain the first rendered image. For example, if the original resolution of the image to be displayed is 4K, the image components in the first region are rendered with the 4K resolution to obtain the first rendered image.

[0146] In the above step S1202, the second region in the image to be displayed or the image to be displayed is down-sampled, and the second region in the image to be displayed or the image to be displayed after the down-sampling is rendered to obtain a second rendered image.

[0147] As an example, the second region in the image to be displayed is down-sampled, for example, the resolution is reduced from 4K to 1080P, and then the second region in the down-sampled image to be displayed is rendered to obtain a second rendered image.

[0148] As another example, the image to be displayed is directly downsampled, for example, the resolution is reduced from 4K to 1080P, and the downsampled image to be displayed is rendered to obtain a second rendered image.

[0149] The above solution ensures the user's viewing experience by performing high-precision rendering on the first area corresponding to the first gaze position, and performs downsampling and low-resolution rendering on the second area outside the first area to reduce device performance pressure and improve image rendering efficiency.

[0150] As an example, taking rectangular segmentation as an example, the image to be displayed is evenly divided into nine 3*3 areas. When the human eye is looking at a certain area, the area is rendered with high precision, and when the gaze point is not inside the area, it is rendered with low precision. This method can improve efficiency by about 70%. For example, when a 4K image is downsampled to 1080p, this solution only needs to render pixels, and a 4K image has a total of 8,847,360 pixels, so only 34.5% of the pixels need to be rendered.

[0151] As another example, taking circular segmentation as an example, downsampling is performed by superimposing a small circular high-precision area on a low-resolution image. The center of the circular area is the position of the human eye's gaze point, and the area needs to change its position at any time to follow the gaze point. Compared with fixed division, this solution can render a smaller screen area, which can further improve performance.

[0152] In an exemplary embodiment, in order to achieve the following between the first rendered image and the second rendered image in the first target image: Fig.13 As shown, step S1206 may include: step S1300, performing feathering processing on the edge of the first rendered image; step S1302, superimposing the feathered first rendered image and the second rendered image to obtain the first target image.

[0153] Through the above scheme, a feathering effect is used at the edge of the high-precision image on the upper layer to be superimposed with the downsampled image on the lower layer, and the two images are smoothly mixed by fusion, thereby preventing a strong sense of tearing at the junction of the high-precision and low-precision images.

[0154] Since the gaze position of the human eye needs to be calculated, it cannot be completely real-time and there will still be a certain degree of hysteresis. Therefore, in an exemplary embodiment, it is necessary to predict the gaze position to prevent the actual high-precision rendering area from not being able to keep up with the changes in the user's gaze position, affecting the user's viewing experience. Fig.14 As shown, the method also includes: step S1400, determining the motion parameters of the first gaze position according to a plurality of consecutive first gaze positions of the target user on the display interface before the current moment; step S1402, determining the second gaze position of the target user at the current moment according to the motion parameters of the first gaze position and a plurality of consecutive first gaze positions before the current moment; step S1404, when the first gaze position of the target user at the current moment is not determined, determining a third area in the image to be displayed corresponding to the second gaze position and a fourth area in the image to be displayed excluding the third area according to the second gaze position; step S1406, rendering the third area and the fourth area in the image to be displayed respectively to obtain a third rendered image and a fourth rendered image, and merging the third rendered image and the fourth rendered image to obtain a second target image, wherein the image quality of the third rendered image is higher than that of the fourth rendered image.

[0155] In the above step S1400, the multiple consecutive first gaze positions of the target user on the display interface before the current moment are determined based on the multiple consecutive frames of eye images before the current moment, each frame of eye image corresponds to a first gaze position, and the motion parameters of the first gaze position, such as the motion speed and acceleration of the first position, can be determined based on the multiple consecutive first gaze positions.

[0156] In the above step S1402, after the motion parameter of the first position is determined, the second gaze position at the current moment can be obtained by making corrections based on a plurality of consecutive first gaze positions according to the motion parameter.

[0157] Exemplarily, taking three consecutive first gaze positions as an example, the second gaze position at the current moment can be determined by the following formula:

[0158]

[0159] Wherein, P is the second gaze position at the current moment, P0, P1, P2 are three consecutive first gaze positions before the current moment (corresponding to three consecutive frames of eye images), and t is the unit time, which is 1 here.

[0160] In the above steps S1404-S1406, if the first gaze position of the target user on the display interface at the current moment has not been calculated, the image to be displayed is rendered according to the second gaze position at the current moment inferred according to multiple consecutive first gaze positions before the current moment in steps S1400-S1402. The rendering method is similar to that when rendering according to the first gaze position, and will not be repeated here.

[0161] In an exemplary embodiment, the eye image is acquired through a monocular camera. That is, the human eye tracking is realized through the monocular camera, so that the displayed image is locally rendered with high precision according to the gaze position of the human eye, while ensuring the user's viewing experience and reducing the performance pressure of the device, the cost is reduced, and no complex and expensive eye tracking equipment is required.

[0162] Embodiment 2

[0163] Fig.15 The block diagram of the image processing device according to the second embodiment of the present application is schematically shown. The image processing device can be divided into one or more program modules, and one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment.

[0164] like Fig.15 As shown, the image processing device 1500 may include an eye image acquisition module 1510 , a first gaze position determination module 1520 , a region determination module 1530 and a rendering module 1540 .

[0165] The eye image acquisition module 1510 is used to acquire the eye image of the target user through a camera, and the eye image is a continuous multi-frame image.

[0166] The first gaze position determining module 1520 is configured to determine a first gaze position of the target user on the display interface according to the eye image.

[0167] an area determination module 1530, configured to determine, according to the first gaze position, a first area in the image to be displayed corresponding to the first gaze position, and a second area in the image to be displayed other than the first area.

[0168] The rendering module 1540 is used to render the first area and the second area in the image to be displayed respectively to obtain a first rendered image and a second rendered image, and merge the first rendered image and the second rendered image to obtain a first target image, wherein the image quality of the first rendered image is higher than that of the second rendered image.

[0169] In an exemplary embodiment, the eye image acquisition module 1510 is used to: acquire a facial image of the target user; binarize the facial image using at least two different thresholds to obtain at least two binary facial images; and determine the eye image of the target user based on the at least two binary facial images.

[0170] In an exemplary embodiment, the eye image acquisition module 1510 is further used to: determine the pupil area of ​​the human eye based on the circular areas in at least two of the binary face images; and determine the eye image based on the pupil area of ​​the human eye.

[0171] In an exemplary embodiment, the first gaze position determination module 1520 is used to: use a pre-trained gaze position prediction model to process the eye image of the current frame to obtain the pupil movement offset of the target user in the current frame; determine the initial predicted gaze position of the target user in the current frame according to the initial predicted gaze position of the target user in the previous frame and the pupil movement offset of the target user in the current frame; determine the first gaze position of the target user in the current frame according to the initial predicted gaze position of the target user in the current frame and the predetermined eye states of the target user when viewing multiple preset calibration points on the display interface.

[0172] In an exemplary embodiment, the first gaze position determination module 1520 is also used to: determine the eye state of the target user in the current frame based on the eye image of the target user in the current frame; determine at least three preset calibration points adjacent to the initial predicted gaze position based on the eye state of the target user in the current frame and the eye states of the target user predetermined when viewing a plurality of preset calibration points on the display interface; determine the first gaze position based on the initial predicted gaze position and at least three preset calibration points adjacent to the initial predicted gaze position.

[0173] In an exemplary embodiment, the first gaze position determination module 1520 is also used to: adjust the initial predicted gaze position of the target user in the current frame, and use the adjusted initial predicted gaze position as the first gaze position when the sum of the distances between the adjusted initial predicted gaze position and the at least three preset calibration points reaches the minimum.

[0174] In an exemplary embodiment, the eye image acquisition module 1510 is also used to: acquire the eye image of the target user when the preset calibration point on the display interface is the gaze position; and determine the eye state of the target user when viewing the preset calibration point on the display interface based on the eye image of the target user when the preset calibration point on the display interface is the gaze position.

[0175] In an exemplary embodiment, the first gaze position determination module 1520 is also used to determine the initial predicted gaze position of the target user in the first frame based on the predetermined eye state of the target user when viewing multiple preset calibration points on the display interface and the eye image of the target user in the first frame.

[0176] In an exemplary embodiment, the region determination module 1530 is used to: divide the image to be displayed into multiple rectangular regions, and use the region including the first gaze position as the first region, and the region excluding the first gaze position as the second region.

[0177] In an exemplary embodiment, the region determination module 1530 is used to: use a circular region with a certain radius and the first gaze position as the center in the image to be displayed as the first region, and use a region outside the first region as the second region.

[0178] In an exemplary embodiment, the rendering module 1540 is used to: downsample the second region in the image to be displayed or the image to be displayed, and render the downsampled second region in the image to be displayed or the image to be displayed to obtain a second rendered image.

[0179] In an exemplary embodiment, the rendering module 1540 is used to: perform feathering processing on the edge of the first rendered image; and superimpose the feathered first rendered image with the second rendered image to obtain the first target image.

[0180] In an exemplary embodiment, the device also includes a second gaze position determination module (not shown), which is used to: determine the motion parameters of the first gaze position based on a plurality of consecutive first gaze positions of the target user on the display interface before the current moment; determine the second gaze position of the target user at the current moment based on the motion parameters of the first gaze position and a plurality of consecutive first gaze positions before the current moment; the area determination module 1530 is also used to: when the first gaze position of the target user at the current moment is not determined, determine a third area in the image to be displayed corresponding to the second gaze position and a fourth area in the image to be displayed excluding the third area according to the second gaze position; the rendering module 1540 is also used to: render the third area and the fourth area in the image to be displayed respectively to obtain a third rendered image and a fourth rendered image, and merge the third rendered image and the fourth rendered image to obtain a second target image, wherein the image quality of the third rendered image is higher than that of the fourth rendered image.

[0181] In an exemplary embodiment, the camera is a monocular camera.

[0182] Embodiment 3

[0183] Fig.16 The schematic diagram of the hardware architecture of a computer device 10000 suitable for the image processing method according to the third embodiment of the present application is shown schematically. The computer device 10000 can be used as a live broadcast server, or a live broadcast terminal, or a part of a live broadcast server or a live broadcast terminal. The computer device 10000 can be a device that can automatically perform numerical calculations and / or data processing according to pre-set or stored instructions. For example, it can be a rack server, a blade server, a tower server or a cabinet server (including an independent server, or a server cluster composed of multiple servers), a gateway, etc. Fig.13 As shown, the computer device 10000 includes at least but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them:

[0184] The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as a hard disk or a memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, referred to as SMC), a secure digital (Secure Digital, referred to as SD) card, a flash card, etc. Of course, the memory 10010 may also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store an operating system and various application software installed in the computer device 10000, such as program codes of image processing methods, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or are to be output.

[0185] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0186] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0187] It should be pointed out that Fig.16 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may be implemented instead.

[0188] In this embodiment, the image processing method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (processor 10020 in this embodiment) to complete the embodiment of the present application.

[0189] Embodiment 4

[0190] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image processing method in the embodiment are implemented.

[0191] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, referred to as SMC), a secure digital (Secure Digital, referred to as SD) card, a flash card, etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the image processing method in the embodiment, etc. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0192] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0193] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an eye image of a target user, wherein the eye image is a continuous multi-frame image; Determine a first gaze position of the target user on the display interface according to the eye image; Determine, according to the first gaze position, a first area in the image to be displayed corresponding to the first gaze position, and a second area in the image to be displayed except the first area; Rendering the first region and the second region in the image to be displayed respectively to obtain a first rendered image and a second rendered image, and merging the first rendered image and the second rendered image to obtain a first target image, wherein the image quality of the first rendered image is higher than that of the second rendered image; Wherein, determining a first gaze position of the target user on the display interface according to the eye image includes: Processing the eye image of the current frame using a pre-trained gaze position prediction model to obtain the pupil movement offset of the target user's eye in the current frame; Determine the initial predicted gaze position of the target user in the current frame according to the initial predicted gaze position of the target user in the previous frame and the pupil movement offset of the target user in the current frame; Determining the first gaze position of the target user in the current frame according to an initial predicted gaze position of the target user in the current frame and a predetermined eye state of the target user when viewing a plurality of preset calibration points on the display interface; Wherein, determining the first gaze position of the target user in the current frame according to the initial predicted gaze position of the target user in the current frame and the predetermined eye states of the target user when viewing a plurality of preset calibration points on the display interface includes: Determining the eye state of the target user in a current frame according to the eye image of the target user in a current frame; Determine at least three preset calibration points adjacent to the initial predicted gaze position according to the eye state of the target user in a current frame and the predetermined eye states of the target user when viewing a plurality of preset calibration points on the display interface; The first gaze position is determined according to the initial predicted gaze position and at least three of the preset calibration points adjacent to the initial predicted gaze position.

2. The image processing method according to claim 1, characterized in that: The step of obtaining the eye image of the target user includes: Acquire a facial image of the target user; Using at least two different thresholds to perform binarization processing on the face image to obtain at least two binarized face images; The eye image of the target user is determined based on at least two of the binary face images.

3. The image processing method according to claim 2, characterized in that: The step of determining the eye image of the target user according to at least two of the binary face images comprises: Determine the pupil area of ​​the human eye according to the circular areas in at least two of the binary face images; The eye image is determined according to the pupil area of ​​the human eye.

4. The image processing method according to claim 1, characterized in that: The determining the first gaze position according to the initial predicted gaze position and at least three preset calibration points adjacent to the initial predicted gaze position includes: The initial predicted gaze position of the target user in the current frame is adjusted, and when the sum of the distances between the adjusted initial predicted gaze position and the at least three preset calibration points reaches a minimum, the adjusted initial predicted gaze position is used as the first gaze position.

5. The image processing method according to claim 1, characterized in that: Before acquiring the eye image of the target user, the method further includes: Acquire an eye image of the target user when the preset calibration point on the display interface is used as the gaze position; The eye state of the target user when viewing the preset calibration point on the display interface is determined according to the eye image of the target user when the preset calibration point on the display interface is the gaze position.

6. The image processing method according to claim 1, characterized in that: The determining a first gaze position of the target user on the display interface according to the eye image further includes: An initial predicted gaze position of the target user in the first frame is determined according to the predetermined eye states of the target user when viewing a plurality of preset calibration points on the display interface and the eye image of the target user in the first frame.

7. The image processing method according to any one of claims 1 to 6, characterized in that: The determining, according to the first gaze position, a first area in the image to be displayed corresponding to the first gaze position, and a second area in the image to be displayed except the first area, comprises: The image to be displayed is divided into a plurality of rectangular areas, and an area including the first gaze position is used as the first area, and an area excluding the first gaze position is used as the second area.

8. The image processing method according to any one of claims 1 to 6, characterized in that: The determining, according to the first gaze position, a first area in the image to be displayed corresponding to the first gaze position, and a second area in the image to be displayed except the first area, comprises: In the image to be displayed, a circular area with a certain radius and the first gaze position as the center is used as the first area, and an area outside the first area is used as the second area.

9. The image processing method according to any one of claims 1 to 6, characterized in that: The rendering of the first region and the second region in the image to be displayed to obtain a first rendered image and a second rendered image respectively includes: Rendering the first region in the image to be displayed with original image quality to obtain the first rendered image; Down-sampling the second region in the image to be displayed or the image to be displayed, and rendering the down-sampled second region in the image to be displayed or the image to be displayed to obtain the second rendered image.

10. The image processing method according to any one of claims 1 to 6, characterized in that: The step of merging the first rendered image and the second rendered image to obtain a first target image includes: Performing feathering processing on the edge of the first rendered image; The first rendered image that has undergone feathering processing is superimposed with the second rendered image to obtain the first target image.

11. The image processing method according to any one of claims 1 to 6, characterized in that: The method further comprises: Determine a motion parameter of the first gaze position according to a plurality of consecutive first gaze positions of the target user on the display interface before a current moment; Determine a second gaze position of the target user at a current moment according to the motion parameter of the first gaze position and a plurality of consecutive first gaze positions before a current moment; When the first gaze position of the target user at the current moment is not determined, determining, according to the second gaze position, a third area in the image to be displayed corresponding to the second gaze position, and a fourth area in the image to be displayed except the third area; The third area and the fourth area in the image to be displayed are rendered respectively to obtain a third rendered image and a fourth rendered image, and the third rendered image and the fourth rendered image are merged to obtain a second target image, wherein the image quality of the third rendered image is higher than that of the fourth rendered image.

12. The image processing method according to any one of claims 1 to 6, characterized in that: The eye image is acquired through a monocular camera.

13. An image processing device, characterized in that: include: An eye image acquisition module is used to acquire an eye image of a target user, wherein the eye image is a continuous multi-frame image; A first gaze position determining module, configured to determine a first gaze position of the target user on the display interface according to the eye image; an area determination module, configured to determine, according to the first gaze position, a first area in the image to be displayed corresponding to the first gaze position, and a second area in the image to be displayed except the first area; a rendering module, configured to render the first region and the second region in the image to be displayed respectively to obtain a first rendered image and a second rendered image, and merge the first rendered image and the second rendered image to obtain a first target image, wherein the image quality of the first rendered image is higher than that of the second rendered image; Wherein, the first gaze position determination module is further used for: Processing the eye image of the current frame using a pre-trained gaze position prediction model to obtain the pupil movement offset of the target user's eye in the current frame; Determine the initial predicted gaze position of the target user in the current frame according to the initial predicted gaze position of the target user in the previous frame and the pupil movement offset of the target user in the current frame; Determining the first gaze position of the target user in the current frame according to an initial predicted gaze position of the target user in the current frame and a predetermined eye state of the target user when viewing a plurality of preset calibration points on the display interface; Wherein, the first gaze position determination module is further used for: Determining the eye state of the target user in a current frame according to the eye image of the target user in a current frame; Determine at least three preset calibration points adjacent to the initial predicted gaze position according to the eye state of the target user in a current frame and the predetermined eye states of the target user when viewing a plurality of preset calibration points on the display interface; The first gaze position is determined according to the initial predicted gaze position and at least three of the preset calibration points adjacent to the initial predicted gaze position.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it is used to implement the steps of the image processing method described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by at least one processor so that the at least one processor executes the steps of the image processing method according to any one of claims 1 to 12.

16. A computer program product, comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 12 are implemented.

Images