Gaze point position prediction model pre-training method and super-resolution model pre-training method

Abstract

This application provides a pre-training method for a gaze point location prediction model and a pre-training method for a super-resolution model. The pre-training method for the gaze point location prediction model includes: acquiring multiple first sample images; determining a first ratio for each image patch in each first sample image, where the first ratio is the ratio of the number of gaze point locations in each image patch to the total number of gaze point locations in the first sample image; sorting the multiple first ratios of each first sample image from largest to smallest to obtain a ratio sequence; obtaining multiple second ratios of each first sample image from the ratio sequence; obtaining multiple first labels for each first sample image; and pre-training the gaze point location prediction model based on the multiple first sample images and the multiple first labels of each first sample image. The technical solution of this application shortens the pre-training time of the gaze point location prediction model and improves the accuracy of the gaze point location prediction model.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a pre-training method for a gaze point location prediction model and a pre-training method for a super-resolution model. Background Technology

[0002] With the continuous maturation of neural network training technology and the continuous improvement of graphics card computing power, its application scope is constantly expanding, especially in the field of image processing, including image processing, image recognition, and image generation. For example, image recognition is based on gaze point location prediction models to identify image patches of interest to users.

[0003] In related technologies, in order to enable image recognition based on a gaze point location prediction model, the gaze point location prediction model is trained using a large amount of randomly acquired training data.

[0004] However, the above-mentioned model training method determines the training data by random selection, which results in the training data lacking specificity. Consequently, the training time for the gaze point location prediction model is long, and the accuracy of the gaze point location prediction model is also poor. Summary of the Invention

[0005] This application provides a pre-training method for a gaze point location prediction model and a super-resolution model, which can obtain effective training data for specific tasks, thereby shortening the pre-training time of the gaze point location prediction model and improving its accuracy. The technical solution is as follows:

[0006] According to a first aspect of the embodiments of this application, a method for pre-training a gaze point location prediction model is provided, the method comprising:

[0007] Acquire multiple first sample images, and determine a first ratio for each image patch in each first sample image, wherein the first ratio is the ratio of the number of fixation point locations in each image patch to the total number of fixation point locations in the first sample image;

[0008] For each of the first sample images, the multiple first ratios are sorted from largest to smallest to obtain a ratio sequence;

[0009] Multiple second ratios are obtained for each of the first sample images from the ratio sequence, and the multiple second ratios are located in the first preset number of positions in the ratio sequence;

[0010] Obtain multiple first labels for each first sample image, the first label including the coordinates of a first gaze image patch and the second ratio of the first gaze image patch, the first gaze image patch being used to indicate the image patch corresponding to the second ratio;

[0011] The gaze point location prediction model is pre-trained based on multiple first sample images and multiple first labels for each first sample image.

[0012] In one possible implementation, the fixation point location prediction model is pre-trained based on multiple first sample images and multiple first labels for each first sample image, including:

[0013] In the i-th iteration, the gaze point location prediction model is used to predict the gaze point location of the i-th first sample image to obtain multiple second labels for the i-th first sample image. The second labels include the coordinates of multiple second gaze image patches and the third ratio of the second gaze image patch. The third ratio is used to indicate the number of gaze point locations in the second gaze image patch, where i is an integer greater than or equal to 1.

[0014] Obtain the i-th error value, which is used to indicate the ratio difference between each of the first gaze image blocks and the second gaze image blocks with the same coordinates;

[0015] When the i-th iteration meets the preset conditions, pre-training stops.

[0016] In one possible implementation, obtaining the i-th error value includes:

[0017] Determine the second label that has the same coordinates as each of the first labels;

[0018] A plurality of first differences are obtained, the first differences being used to indicate the difference between the second ratio and the third ratio corresponding to the same coordinate; and a plurality of second differences are obtained, the second differences being used to indicate the second ratio that does not correspond to the third ratio corresponding to the same coordinate.

[0019] Summing multiple such differences yields the i-th error value.

[0020] In one possible implementation, after pre-training the gaze location prediction model based on multiple first sample images and multiple first labels for each first sample image, the method further includes:

[0021] Acquire multiple sets of images of the same target captured by a camera group, each set of images including a left-eye image, a right-eye image, and a depth image;

[0022] For each image group, based on the depth image in the image group, the left-eye image and the right-eye image in the image group are processed to obtain the sample left-eye image and the sample right-eye image;

[0023] The left-eye image and right-eye image of each image group are combined into a training set. Based on this training set, the fixation point location prediction model is adjusted.

[0024] In one possible implementation, for each image group, the left-eye and right-eye images in the image group are processed based on the depth image in the image group to obtain sample left-eye and sample right-eye images, including:

[0025] For each image group, based on the depth image in the image group, determine the fixation point position of the left eye and the fixation point position of the right eye in the depth image;

[0026] Based on the fixation point positions of the left and right eyes in the depth image, the left and right eye images of the image group are processed respectively to obtain the middle left and right eye images of the image group.

[0027] Upsample the left-eye and right-eye center images of the image group to obtain the sample left-eye and sample right-eye images of the image group.

[0028] According to a second aspect of the embodiments of this application, a pre-training method for a super-resolution model is provided, the method comprising:

[0029] Multiple second sample images are acquired, and a target gaze image patch is obtained for each second sample image based on the gaze point location prediction model. Each second sample image includes multiple image patches, and the target gaze image patch is the image patch in the second sample image that includes the most gaze point locations.

[0030] Based on each target gaze image block, the corresponding second sample image is cropped to obtain multiple first sample sub-images. Each first sample sub-image includes the target gaze image block, and the size of the first sample sub-image is smaller than the size of the second sample image.

[0031] A training dataset is generated based on multiple sub-images of the first sample.

[0032] Based on this training dataset, the super-resolution model is pre-trained.

[0033] In one possible implementation, the gaze location prediction model yields the target gaze image patch for each of the second sample images, including:

[0034] Based on the gaze point location prediction model, gaze point location is predicted for each of the second sample images to obtain multiple labels for each of the second sample images. Each label includes the coordinates of the gaze image block and the ratio of the gaze image block. The ratio is the ratio of the number of gaze point locations in each gaze image block to the total number of gaze point locations in the second sample image. The gaze image block is the image block corresponding to the first preset number of positions in the ratio sequence. The ratio sequence is a sequence in which the ratios are arranged from smallest to largest.

[0035] Obtain the target ratio for each of the second sample images, where the target ratio is the largest ratio in the ratio sequence.

[0036] The gaze image block corresponding to the target ratio is taken as the target gaze image block.

[0037] In one possible implementation, the second sample image is cropped based on each target gaze image block to obtain multiple first sample sub-images, including:

[0038] Centered on each target gaze image block, the first sample sub-image is cropped from the corresponding second sample image according to the preset side length, and the remaining part is output as the second sample sub-image.

[0039] In one possible implementation, the training dataset is generated based on multiple of the first sample sub-images, including:

[0040] Each of the first sample sub-images is downsampled to obtain multiple downsampled images;

[0041] Each downsampled image is upsampled to obtain multiple upsampled images in the training dataset. Each upsampled image has the same resolution and size as the corresponding first sample sub-image.

[0042] In one possible implementation, pre-training the super-resolution model based on the training dataset includes:

[0043] In the k-th iteration, the k-th first sample sub-image is input into the super-resolution model, outputting the k-th first supramolecular image. The k-th first sample sub-image is then input into a pre-defined supervised model, outputting the k-th second supramolecular image. The k-th first loss value and the k-th second loss value are obtained. The k-th first loss value indicates the information difference between the k-th first supramolecular image and the k-th second supramolecular image, and the k-th second loss value indicates the information difference between the k-th second supramolecular image and the k-th first sample sub-image. k is an integer greater than or equal to 1.

[0044] Based on the kth first loss value and the kth second loss value, determine the loss value and the value;

[0045] Based on this loss value and value, the model parameters of the super-resolution model are adjusted.

[0046] In one possible implementation, the pre-training of the super-resolution model based on the training dataset also includes:

[0047] During the k-th iteration, the k-th third loss value is obtained. This k-th third loss value is used to indicate the information difference between the k-th first supramolecular image and the k-th first sample sub-image.

[0048] Based on the k-th first loss value and the k-th second loss value, the loss value and the value are determined as follows:

[0049] The sum of the k-th first loss value, the k-th second loss value, and the k-th third loss value is calculated to obtain the sum of the loss values.

[0050] In one possible implementation, after pre-training the super-resolution model based on the training dataset, the method further includes:

[0051] Acquire multiple sets of images of the same target captured by a camera group, each set of images including a left-eye image, a right-eye image, and a depth image;

[0052] For each image group, based on the depth image in the image group, the left-eye image and the right-eye image in the image group are processed to obtain the sample left-eye image and the sample right-eye image;

[0053] The left-eye image and right-eye image of each sample corresponding to the image group are combined into a training set, and the super-resolution model is adjusted based on the training set.

[0054] In one possible implementation, for each image group, the left-eye and right-eye images in the image group are processed based on the depth image in the image group to obtain sample left-eye and sample right-eye images, including:

[0055] For each image group, based on the depth image in the image group, determine the fixation point position of the left eye and the fixation point position of the right eye in the depth image;

[0056] Based on the fixation point positions of the left and right eyes in the depth image, the left and right eye images of the image group are processed respectively to obtain the middle left and right eye images of the image group.

[0057] Upsample the left-eye and right-eye center images of the image group to obtain the sample left-eye and sample right-eye images of the image group.

[0058] In one possible implementation, the super-resolution model is adjusted based on the training set, including:

[0059] Based on the fixation point location prediction model, the left-eye target fixation image block of each left-eye image of the sample and the right-eye target fixation image block of each right-eye image of the sample are obtained in the training set. Each left-eye image of the sample includes multiple left-eye image blocks, and the left-eye target fixation image block is the left-eye image block with the most fixation point locations in the left-eye image of the sample. Each right-eye image of the sample includes multiple right-eye image blocks, and the right-eye target fixation image block is the right-eye image block with the most fixation point locations in the right-eye image of the sample.

[0060] Based on each left-eye gaze image block of the target, the first sample left-eye sub-image is cropped from the corresponding sample left-eye image; based on each right-eye gaze image block of the target, the first sample right-eye sub-image is cropped from the corresponding sample right-eye image.

[0061] The super-resolution model is adjusted based on multiple left-eye images and multiple right-eye images of the first sample.

[0062] According to a third aspect of the embodiments of this application, an image processing method is provided, the method comprising:

[0063] The images acquired by the camera group are processed to obtain two initial images;

[0064] Based on the gaze point location prediction model, gaze point location is predicted for each initial image to obtain the target gaze image block for each initial image. Each initial image includes multiple image blocks, and the target gaze image block is the image block in the initial image that contains the most gaze point locations.

[0065] Based on the target gaze image block, a first initial sub-image is cropped from each of the initial images, and the remaining part is the second initial sub-image. The size of the first initial sub-image is smaller than the size of the initial image.

[0066] Based on the super-resolution model, each of the first initial sub-images is super-resolutiond to obtain two first target sub-images;

[0067] Each of the first target sub-images is combined with its corresponding second initial sub-image to obtain two target images.

[0068] According to a fourth aspect of the embodiments of this application, a pre-training apparatus for a gaze point location prediction model is provided, the apparatus comprising:

[0069] The first acquisition module is used to acquire multiple first sample images and determine a first ratio of each image block in each first sample image. The first ratio is the ratio of the number of fixation point positions in each image block to the total number of fixation point positions in the first sample image.

[0070] The sorting module is used to sort the multiple first ratios of each first sample image from largest to smallest to obtain a ratio sequence;

[0071] The second acquisition module is used to acquire multiple second ratios for each of the first sample images from the ratio sequence, wherein the multiple second ratios are located in the first preset number of positions in the ratio sequence;

[0072] The third acquisition module is used to acquire multiple first labels for each of the first sample images. The first label includes the coordinates of the first gaze image block and the second ratio of the first gaze image block. The first gaze image block is used to indicate the image block corresponding to the second ratio.

[0073] The pre-training module is used to pre-train the gaze point location prediction model based on multiple first sample images and multiple first labels for each first sample image.

[0074] In one possible implementation, the pre-training module includes:

[0075] An iterative unit is used to predict the gaze position of the i-th first sample image through the gaze position prediction model during the i-th iteration, and obtain multiple second labels for the i-th first sample image. The second labels include the coordinates of multiple second gaze image blocks and the third ratio of the second gaze image block. The third ratio is used to indicate the number of gaze positions in the second gaze image block, where i is an integer greater than or equal to 1.

[0076] An acquisition unit is used to acquire the i-th error value, which is used to indicate the ratio difference between each of the first gaze image blocks and the second gaze image block with the same coordinates;

[0077] The judgment unit is used to stop pre-training when the i-th iteration process meets the preset conditions.

[0078] In one possible implementation, the acquiring unit includes:

[0079] Determine sub-units to respectively determine the second label that has the same coordinates as each of the first labels;

[0080] The sub-unit is configured to acquire multiple first differences, which are used to indicate the difference between the second ratio and the third ratio corresponding to the same coordinate, and to acquire multiple second differences, which are used to indicate the second ratio that does not correspond to the third ratio at the same coordinate.

[0081] The calculation subunit is used to sum multiple such differences to obtain the i-th error value.

[0082] In one possible implementation, the device further includes:

[0083] The acquisition module is used to acquire multiple image groups of the same target acquired by the camera group. Each image group includes a left-eye image, a right-eye image, and a depth image.

[0084] The processing module is used to process the left-eye and right-eye images in each image group based on the depth image in the image group, so as to obtain sample left-eye and sample right-eye images.

[0085] The adjustment module is used to form a training set by combining the sample left-eye image and sample right-eye image corresponding to each image group, and to adjust the gaze point position prediction model based on the training set.

[0086] In one possible implementation, the processing module includes:

[0087] The determining unit is used to determine, for each image group, the fixation point position of the left eye and the fixation point position of the right eye in the depth image of the image group.

[0088] The processing unit is used to process the left-eye image and the right-eye image of the image group according to the fixation point position of the left eye and the fixation point position of the right eye in the depth image, respectively, to obtain the left-eye middle image and the right-eye middle image of the image group.

[0089] The upsampling unit is used to upsample the left-eye middle image and the right-eye middle image of the image group to obtain the sample left-eye image and sample right-eye image of the image group.

[0090] According to a fifth aspect of the embodiments of this application, a super-resolution model pre-training apparatus is provided, the apparatus comprising:

[0091] The acquisition module is used to acquire multiple second sample images and obtain a target gaze image block for each second sample image based on the gaze point location prediction model. Each second sample image includes multiple image blocks, and the target gaze image block is the image block in the second sample image that includes the most gaze point locations.

[0092] The cropping module is used to crop the corresponding second sample image based on each target gaze image block to obtain multiple first sample sub-images. Each first sample sub-image includes the target gaze image block, and the size of the first sample sub-image is smaller than the size of the second sample image.

[0093] The generation module is used to generate a training dataset based on multiple of the first sample sub-images;

[0094] The pre-training module is used to pre-train the super-resolution model based on the training dataset.

[0095] In one possible implementation, the acquisition module includes:

[0096] The prediction unit is used to predict the gaze position of each second sample image based on the gaze position prediction model, and obtain multiple labels for each second sample image. Each label includes the coordinates of the gaze image block and the ratio of the gaze image block. The ratio is the ratio of the number of gaze positions in each gaze image block to the total number of gaze positions in the second sample image. The gaze image block is the image block corresponding to the first preset number of positions in the ratio sequence. The ratio sequence is a sequence in which the ratios are arranged from smallest to largest.

[0097] An acquisition unit is used to acquire the target ratio of each of the second sample images, wherein the target ratio is the largest ratio in the ratio sequence;

[0098] The first determining unit is used to determine the gaze image block corresponding to the target ratio as the target gaze image block.

[0099] In one possible implementation, the trimming module includes:

[0100] The first cropping unit is used to crop the first sample sub-image from the corresponding second sample image based on a preset side length, with each target gaze image block as the center, and output the remaining part as the second sample sub-image.

[0101] In one possible implementation, the generation module includes:

[0102] The downsampling unit is used to downsample each of the first sample sub-images to obtain multiple downsampled images;

[0103] The first upsampling unit is used to upsample each of the downsampled images to obtain multiple upsampled images included in the training dataset. The resolution and size of the upsampled images are the same as those of the corresponding first sample sub-images.

[0104] In one possible implementation, the pre-training module includes:

[0105] The first iteration unit is used to, during the k-th iteration, input the k-th first sample sub-image into the super-resolution model and output the k-th first supramolecular image; input the k-th first sample sub-image into a preset supervised model and output the k-th second supramolecular image; and obtain the k-th first loss value and the k-th second loss value. The k-th first loss value is used to indicate the information difference between the k-th first supramolecular image and the k-th second supramolecular image, and the k-th second loss value is used to indicate the information difference between the k-th second supramolecular image and the k-th first sample sub-image, where k is an integer greater than or equal to 1.

[0106] The first calculation unit is used to determine the loss value and the value based on the kth first loss value and the kth second loss value;

[0107] The adjustment unit is used to adjust the model parameters of the super-resolution model based on the loss value and the value.

[0108] In one possible implementation, the pre-training module further includes:

[0109] The second iteration unit is used to obtain the k-th third loss value during the k-th iteration. The k-th third loss value is used to indicate the information difference between the k-th first supramolecular image and the k-th first sample sub-image.

[0110] The first computing unit includes:

[0111] The calculation subunit is used to sum the k-th first loss value, the k-th second loss value, and the k-th third loss value to obtain the sum of the loss values.

[0112] In one possible implementation, the device further includes:

[0113] The acquisition module is used to acquire multiple image groups of the same target acquired by the camera group. Each image group includes a left-eye image, a right-eye image, and a depth image.

[0114] The processing module is used to process the left-eye and right-eye images in each image group based on the depth image in the image group, so as to obtain sample left-eye and sample right-eye images.

[0115] The adjustment module is used to form a training set by combining the left-eye image and the right-eye image of each sample in the image group, and to adjust the super-resolution model based on the training set.

[0116] In one possible implementation, the processing module includes:

[0117] The second determining unit is used to determine, for each image group, the fixation point position of the left eye and the fixation point position of the right eye in the depth image of the image group.

[0118] The processing unit is used to process the left-eye image and the right-eye image of the image group according to the fixation point position of the left eye and the fixation point position of the right eye in the depth image, respectively, to obtain the left-eye middle image and the right-eye middle image of the image group.

[0119] The second upsampling unit is used to upsample the left-eye middle image and the right-eye middle image of the image group to obtain the sample left-eye image and sample right-eye image of the image group.

[0120] In one possible implementation, the adjustment module includes:

[0121] The second prediction unit is used to obtain the left-eye target gaze image block of each left-eye image of the sample and the right-eye target gaze image block of each right-eye image of the sample based on the gaze point location prediction model. Each left-eye image of the sample includes multiple left-eye image blocks, and the left-eye target gaze image block is the left-eye image block with the most gaze point locations in the left-eye image of the sample. Each right-eye image of the sample includes multiple right-eye image blocks, and the right-eye target gaze image block is the right-eye image block with the most gaze point locations in the right-eye image of the sample.

[0122] The second cropping unit is used to crop out a first sample left eye sub-image from the corresponding sample left eye image for each left eye gaze image block of the target; and to crop out a first sample right eye sub-image from the corresponding sample right eye image for each right eye gaze image block of the target.

[0123] The adjustment unit is used to adjust the super-resolution model based on multiple left-eye images and multiple right-eye images of the first sample.

[0124] According to a sixth aspect of the embodiments of this application, an image processing apparatus is provided, the apparatus comprising:

[0125] The processing module is used to process the images acquired by the camera group to obtain two initial images;

[0126] The prediction module is used to predict the gaze point location for each initial image based on the gaze point location prediction model, and obtain the target gaze image block for each initial image. Each initial image includes multiple image blocks, and the target gaze image block is the image block in the initial image that contains the most gaze point locations.

[0127] The cropping module is used to crop a first initial sub-image from each of the initial images based on the target gaze image block, with the remaining part being a second initial sub-image, the size of which is smaller than the size of the initial image;

[0128] The super-resolution module is used to perform super-resolution on each of the first initial sub-images based on the super-resolution model to obtain two first target sub-images;

[0129] The combination module is used to combine each of the first target sub-images with the corresponding second initial sub-image to obtain two target images.

[0130] According to a seventh aspect of the present application, a computer device is provided, the computer device including a processor and a memory, the memory being used to store at least one program, the at least one program being loaded by the processor and executed by the processor to perform the above-described gaze position prediction model pre-training method, the above-described super-resolution model pre-training method, or the above-described image processing method.

[0131] According to an eighth aspect of the present application, a computer-readable storage medium is provided, which stores at least one program that is loaded and executed by a processor to implement the above-described gaze position prediction model pre-training method, the above-described super-resolution model pre-training method, or the above-described image processing method.

[0132] According to an eighth aspect of the embodiments of this application, a computer program product is provided, the computer program product comprising at least one program segment loaded and executed by a processor to implement the gaze position prediction model pre-training method, the super-resolution model pre-training method, or the image processing method described above.

[0133] The beneficial effects of the technical solutions provided in this application are:

[0134] In this embodiment of the application, a pre-training method for a gaze point location prediction model is provided. By determining the label of each first sample image, invalid training data is eliminated, and effective training data for a specific task is obtained, thereby shortening the pre-training time of the gaze point location prediction model and improving the accuracy of the gaze point location prediction model. Attached Figure Description

[0135] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0136] Figure 1 This is a schematic diagram of an implementation environment provided according to an embodiment of this application;

[0137] Figure 2 This is a flowchart illustrating a pre-training method for a gaze point position prediction model according to an embodiment of this application;

[0138] Figure 3 This is an example flowchart illustrating a pre-training method for a gaze point location prediction model provided in an embodiment of this application;

[0139] Figure 4 This is a schematic diagram of a face image provided according to an embodiment of this application;

[0140] Figure 5 This is a schematic diagram illustrating the principle of determining the center of the eyeball according to an embodiment of this application;

[0141] Figure 6 This is a schematic diagram illustrating the principle of determining the position of the gaze point according to an embodiment of this application;

[0142] Figure 7 This is a schematic diagram of the positions of various image blocks and gaze points in a sample image provided according to an embodiment of this application;

[0143] Figure 8 This is a flowchart illustrating a super-resolution model pre-training method according to an embodiment of this application;

[0144] Figure 9 This is a schematic flowchart of an example of a super-resolution model pre-training method provided in an embodiment of this application;

[0145] Figure 10 This is a schematic flowchart of an image processing method provided according to an embodiment of this application;

[0146] Figure 11 This is a schematic diagram showing the positions of various cameras on the screen according to an embodiment of this application;

[0147] Figure 12 This is a schematic diagram of a human eye and its corresponding gaze point according to an embodiment of this application;

[0148] Figure 13 This is a schematic flowchart of an image processing method provided according to an embodiment of this application;

[0149] Figure 14 This is a schematic diagram of the structure of a pre-training device for a gaze point position prediction model according to an embodiment of this application;

[0150] Figure 15This is a schematic diagram of the structure of a super-resolution model pre-training device according to an embodiment of this application;

[0151] Figure 16 This is a schematic diagram of the structure of an image processing apparatus according to an embodiment of this application;

[0152] Figure 17 This is a schematic diagram of the structure of a terminal according to an embodiment of this application;

[0153] Figure 18 This is a schematic diagram of the structure of a server according to an embodiment of this application. Detailed Implementation

[0154] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0155] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0156] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms.

[0157] These terms are simply used to distinguish one element from another. For example, without departing from the scope of various examples, the first action can be called the second action, and similarly, the second action can be called the first action. Both the first and second actions can be actions, and in some cases, they can be separate and distinct actions.

[0158] "At least one" refers to one or more actions. For example, at least one action can be one action, two actions, three actions, or any integer number of actions greater than or equal to one. "Multiple" refers to two or more actions. For example, multiple actions can be two actions, three actions, or any integer number of actions greater than or equal to two.

[0159] It should be noted that all data (including but not limited to training data and prediction data, such as user data and terminal-side data) and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the training data involved in this application was obtained with full authorization.

[0160] Figure 1 This is a schematic diagram of an implementation environment provided according to an embodiment of this application. The implementation environment may include a first terminal 101, a server 102, and a second terminal 103.

[0161] Both the first terminal 101 and the second terminal 102 are equipped with camera components, such as a color camera and / or a depth camera set on a computer or a smartphone.

[0162] The first terminal 101 and the second terminal 102 can be electronic devices such as smartphones, wearable devices, personal computers, laptops, tablets, smart TVs, and in-vehicle terminals.

[0163] Server 102 can be a single server, a server cluster consisting of multiple servers, or a cloud processing center.

[0164] The first terminal 102 and the second terminal 103 are connected to the server 102 via wired or wireless networks, respectively.

[0165] In some embodiments, the wireless or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but can be any network, including but not limited to Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (MANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, technologies and / or formats, including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Networks (VPNs), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, custom and / or dedicated data communication technologies may be used to replace or supplement the aforementioned data communication technologies.

[0166] Figure 2 This is a flowchart illustrating a pre-training method for a gaze point location prediction model according to an embodiment of this application, as shown below. Figure 2 As shown, this embodiment of the application uses a terminal or server as an example for illustration. The method includes the following steps:

[0167] In step 201, the server acquires multiple first sample images and determines a first ratio for each image block in each first sample image. The first ratio is the ratio of the number of gaze points in each image block to the total number of gaze points in the first sample image.

[0168] In some embodiments, multiple first sample videos are acquired, each first sample video is parsed frame by frame to obtain multiple first sample images, each first sample image is divided into multiple image blocks, and the number of gaze points included in each image block of each first sample image and the total number of gaze points in each first sample image are determined to determine a first ratio of gaze points in each image block of each first sample image.

[0169] In step 202, the server sorts the multiple first ratios of each first sample image from largest to smallest to obtain a ratio sequence.

[0170] In step 203, the server obtains multiple second ratios for each of the first sample images from the ratio sequence, wherein the multiple second ratios are located in the first preset number of positions of the ratio sequence.

[0171] In step 204, the server obtains multiple first labels for each of the first sample images. The first label includes the coordinates of the first gaze image block and the second ratio of the first gaze image block. The first gaze image block is used to indicate the image block corresponding to the second ratio.

[0172] In the embodiments of this application, each image block corresponding to the second ratio and the second ratio are used as the first label of the corresponding first sample image. The image block in each first label is the image block that the user is relatively interested in. That is, effective training data is selected from multiple first sample images, and invalid training data is eliminated, thereby obtaining effective training data for a specific task.

[0173] In step 205, the server pre-trains the gaze point location prediction model based on multiple first sample images and multiple first labels for each first sample image.

[0174] The above embodiments are illustrated using a server as the execution subject, but a terminal can also be used as the execution subject.

[0175] In the embodiments of this application, by determining the label of each first sample image, invalid training data is eliminated, and effective training data for a specific task is obtained, thereby shortening the pre-training time of the gaze point location prediction model and improving the accuracy of the gaze point location prediction model.

[0176] Figure 2 The illustrated embodiment is a simplified flow of an embodiment of this application. Below, based on... Figure 3 The technical solution of this application will be further described. Figure 3 This is an example flowchart illustrating a pre-training method for a gaze point location prediction model according to an embodiment of this application. In this embodiment, it is illustrated using an application on a terminal or server. The method includes the following steps:

[0177] In step 301, the server acquires multiple sample videos, parses each sample video frame by frame to obtain a sample video frame sequence, plays the sample video frame sequence to each user, and records the gaze point position of each user on each first sample image in the sample video frame sequence.

[0178] In some embodiments, during the playback of the sample video frame sequence to a user, the user's gaze point position in each of the first sample images is recorded based on a gaze point position prediction algorithm. Recording the user's gaze point position in the first sample image based on the gaze point position prediction algorithm includes the following steps 301A to 301D:

[0179] In step 301A, the server performs face detection on the user to obtain the user's face image.

[0180] In some embodiments, the .XML file in the OpenCV (cross-platform computer vision library) directory is directly called to perform face detection on the user, resulting in... Figure 4 The image contains a face. The .XML file includes a classification cascade and Haar features. Embodiments of this application do not specifically limit the classification cascade and Haar features.

[0181] In step 301B, the server calculates the pixel coordinates of the top left corner of the ROI (region of interest) image of the left eye and the top left corner of the ROI image of the right eye in the face image, and extracts the ROI images of the user's left eye and right eye from the face image.

[0182] In some embodiments, the Region of Interest (ROI) images of the user's left and right eyes are extracted from the user's facial image to determine the approximate regions where the centers of the left and right eyeballs are located. Based on prior knowledge, the size and position of the ROI images of the left and right eyes in a facial image are generally fixed. For example, as... Figure 4 As shown, let the length of the entire face image be denoted as wl (wl > 0), and the width as hl (hl > 0). Let the length of the eye region be wE (wE > 0), and the width of the eye region be hE (hE > 0). Then wE = 35 / 100wl and hE = 30 / 100hl. The pixel coordinates of the top-left corner of the left eye's ROI image are: yL = 25 / 100hl, xL = 13 / 100wl; the pixel coordinates of the top-left corner of the right eye's ROI image are: yL = 25 / 100hl, xR = wl - wE - 13 / 100wl. Based on the pixel coordinates of the top-left corner of the left eye's ROI image, the ROI image of the left eye is extracted from the face image; similarly, based on the pixel coordinates of the top-left corner of the right eye's ROI image, the ROI image of the right eye is extracted from the face image. The terms "35 / 100", "30 / 100", "25 / 100", and "13 / 100" in the above formulas are all determined based on prior knowledge. The embodiments of this application do not specifically limit the prior knowledge.

[0183] In step 301C, the server obtains the position of the eyeball center in the left eye's ROI image based on the coordinates of each pixel in the left eye's ROI image, and obtains the position of the eyeball center in the right eye's ROI image based on the coordinates of each pixel in the right eye's ROI image.

[0184] In the left eye's ROI image, one point represents the center of the left eyeball, and in the right eye's ROI image, one point represents the center of the right eyeball.

[0185] In some embodiments, for a ROI image, the closer to the center of the eye, the lower the grayscale value. More pixels will have their gradient direction lines intersecting at the center of the eye. Therefore, the pixel with the most intersecting gradient direction lines is the center of the eye. For example, taking the left eye as an example, based on the coordinates of each pixel in the left eye's ROI image, the gradient of each pixel in the left eye's ROI image is calculated. The gradient of each pixel includes magnitude and direction, for example, as shown... Figure 5 As shown, determine the gradient d of any pixel c in the ROI image of the left eye, find a pixel e along the gradient d direction, the gradient of pixel e is v, extend along the opposite direction of v to obtain a ray, and the intersection of multiple rays is the center of the eyeball of the left eye.

[0186] In step 301D, the server determines the user's gaze point location on the first sample image.

[0187] In some embodiments, step 301D above includes steps 301Da to 301Dc:

[0188] In step 301Da, the server determines multiple projection points on the user's right-eye projection retina to determine any one of the first pair of polar lines on the right-eye projection retina.

[0189] In some embodiments, step 301Da includes determining a first reference point in a world coordinate system, connecting the first reference point with the center of the left eye to obtain a first line, connecting the first reference point with the center of the right eye to obtain a second line, determining the intersection of the second line with the projection retina of the right eye as a first projection point, determining a second reference point on the first line, connecting the second reference point with the center of the right eye to obtain a third line, determining the intersection of the third line with the projection retina of the right eye as a second projection point, and connecting the first projection point and the second projection point to obtain a first polar line on the projection retina of the right eye.

[0190] In step 301Db, the server determines multiple projection points on the user's left-eye projection retina to determine a second pair of epipolar lines on the left-eye projection retina, which matches any one of the first pair of epipolar lines. The projection of any point in the world coordinate system falls on any one of the first pair of epipolar lines on the right-eye projection retina, and the projection of this point also falls on the second pair of epipolar lines on the left-eye projection retina.

[0191] In some embodiments, a first reference point is determined in the world coordinate system, and a first line is obtained by connecting the first reference point with the position of the center of the left eyeball. The intersection of the first line with the projection retina of the left eye is determined as the third projection point. A third reference point is determined on the second line, and a fourth line is obtained by connecting the third reference point with the position of the center of the left eyeball. The intersection of the fourth line with the projection retina of the left eye is determined as the fourth projection point. A second pair of polar lines on the projection retina of the left eye is obtained by connecting the third projection point and the fourth projection point.

[0192] In step 301Dc, the server obtains a first target projection point on the user's left eye projection retina, determines the second pair of epipolar lines corresponding to the projection point, determines a first pair of epipolar lines matching the second pair of epipolar lines on the user's right eye projection retina, determines a second target projection point matching the first target projection point on the second pair of epipolar lines, connects the center of the right eyeball with the first target projection point to obtain a first target line, connects the center of the left eyeball with the second target projection point to obtain a second target line, and the intersection of the first target line and the second target line is the user's fixation point position on the sample image.

[0193] In some embodiments, the user's gaze point location on the sample image is determined based on a stereo ranging method. For example, such as Figure 6As shown, taking any reference point P in the world coordinate system as an example, the center of the left eyeball is located at O1, and the center of the right eyeball is located at O2. Connect P to O1 and O2 respectively. Let f1 be the projection retina of the left eye and f2 be the projection retina of the right eye. The projection of point P onto f1 is p1, and the projection of point P onto f2 is p2. Taking any point X in space as an example, connect X to O1 and O2 respectively. The projection of point X onto f1 is p1, and the projection of point X onto f2 is x2. Taking any point Z in space as an example, connect X to O1 and O2 respectively. The projection of Z onto f1 is p1, and the projection of Z onto f2 is z2. By connecting p1, x2, and z2, an epipolar line k2 of f2 is determined. Then, an epipolar line k1 of f1 is determined. k1 matches k2, so that after determining the h point on k1, a point H matching the h point is found on k2. Connect O1h and O2H. The intersection of O1h and O2H is the fixation point position. The spatial position of the fixation point is calculated based on O1, h, O2, and H. The embodiments of this application do not specifically limit the algorithm for calculating the spatial position of the fixation point.

[0194] In step 302, the server divides each first sample image into multiple image blocks and determines a first ratio for each image block in each first sample image. The first ratio is used to indicate the number of gaze points in each image block.

[0195] In some embodiments, taking any first sample image as an example, the length and width of the first sample image are obtained, and the first sample image is divided based on the preset length and preset width of a preset image block to obtain multiple image blocks. For example... Figure 7 As shown, each first sample image is divided into multiple image blocks. For example, the length of each first sample image is denoted as m, and the width as n, where m and n are integers greater than or equal to 1. Each first sample image is divided into m*n image blocks, and each image block is 1*1 in size. Taking one image block of a first sample image as an example, the number of fixation points included in that image block is obtained, and the ratio of the number of fixation points included in that image block to the total number of fixation points included in the sample image is calculated, which is the first ratio of that image block. For example, if m*n = 100, any first sample image includes 100 fixation points, and the first image block includes 5 fixation points, then the first ratio of the first image block is 0.05.

[0196] In step 303, the server sorts the multiple first ratios of each first sample image from largest to smallest to obtain a ratio sequence.

[0197] In step 304, the server obtains multiple second ratios for each of the first sample images from the ratio sequence, wherein the multiple second ratios are located in the first preset number of positions of the ratio sequence.

[0198] In some embodiments, each first sample image is divided into 100 image blocks, and the first ratio of each image block is calculated to obtain 100 first ratios for each first sample image. The 100 first ratios of each first sample image are sorted from largest to smallest to obtain a ratio sequence for each first sample image. The top 10 first ratios with larger values ​​are obtained from the ratio sequence for each first sample image to obtain multiple second ratios for each first sample image.

[0199] In step 305, the server obtains multiple first labels for each first sample image. The first label includes the coordinates of the first gaze image block and the second ratio of the first gaze image block. The first gaze image block is used to indicate the image block corresponding to the second ratio.

[0200] In some embodiments, the image patch corresponding to the second ratio is defined as the first gaze image patch, and each second ratio and the corresponding first gaze image patch form a first label. For example, any first label of any first sample image is: {0.1, (x, y)}, where x is an integer greater than or equal to 1 and less than or equal to m, y is an integer greater than or equal to 1 and less than or equal to n, (x, y) represents the first gaze image patch in the x-th row and y-th column, and 0.1 is the second ratio of the first gaze image patch.

[0201] In the embodiments of this application, a first ratio of each image block in each first sample image is determined, and a plurality of second ratios with larger values ​​are selected from a plurality of first ratios. The image blocks corresponding to each of the second ratios are determined as first gaze image blocks, that is, valid image blocks, and the image blocks corresponding to the remaining plurality of first ratios with smaller values ​​are determined as invalid image blocks, thereby realizing the distinction between valid image blocks and invalid image blocks in each first sample image.

[0202] In step 306, the server uses multiple first sample images and multiple first labels for each first sample image to pre-train the gaze point location prediction model.

[0203] In some embodiments, step 306 includes, during the i-th iteration, predicting the gaze position of the i-th first sample image using the gaze position prediction model to obtain multiple second labels for the i-th first sample image. The second labels include the coordinates of multiple second gaze image blocks and a third ratio of the second gaze image blocks. The third ratio is used to indicate the number of gaze positions in the second gaze image block, where i is an integer greater than or equal to 1. Obtaining the i-th error value, which is used to indicate the ratio difference between each first gaze image block and a second gaze image block with the same coordinates. Training stops when the i-th iteration meets a preset condition.

[0204] In some embodiments, the third ratios among the plurality of second labels are normalized so that the sum of the plurality of third ratios is 1, in order to facilitate calculation and thus save computational resources.

[0205] In some embodiments, obtaining the i-th error value includes: determining the second label with the same coordinates as each of the first labels; obtaining a plurality of first differences, which indicate the difference between the second ratio and the third ratio corresponding to the same coordinates; and obtaining a plurality of second differences, which indicate the second ratio that does not correspond to the third ratio corresponding to the same coordinates; and summing the plurality of differences to obtain the i-th error value. For example, when i = 6, each sample label includes 5 first labels and 50 image patches. The first first label is: {0.05, (3,6)}; the second first label is: {0.2, (3,7)}; the third first label is: {0.2, (4,6)}; the fourth first label is: {0.2, (4,7)}; and the fifth first label is: {0.3, (4,8)}. The gaze point position prediction model is used to predict the gaze point position of the sixth first sample image to obtain a plurality of second labels for the sixth first sample image. The first second label is {0.1, (3,6)}, the second second label is {0.2, (3,7)}, the third second label is {0.2, (4,9)}, the fourth second label is {0.2, (4,7)}, and the fifth second label is {0.3, (5,6)}. The sixth error value is: |0.05-0.1|+|0.2-0.2|+|0.05-0|+|0.2-0|+|0.2-0.2|+|0.3-0|=0.6. Assuming the preset condition is that the error value tends to 0, the sixth error value does not meet the preset condition, and the seventh iteration continues.

[0206] In step 307, the server acquires multiple image groups of the same target captured by the camera group, each image group including a left-eye image, a right-eye image, and a depth image.

[0207] In some embodiments, the camera group includes a depth camera and two color cameras. For example, the depth camera is used to acquire depth images, one color camera simulates a person's left eye to acquire left-eye images, and the other color camera simulates a person's right eye to acquire right-eye images.

[0208] In step 308, the server processes the left-eye and right-eye images in each image group based on the depth image in that image group to obtain sample left-eye and sample right-eye images.

[0209] In some embodiments, step 308 includes the server determining, for each image group, the fixation point position of the left eye and the fixation point position of the right eye in the depth image of the image group; processing the left eye image and the right eye image of the image group respectively based on the fixation point position of the left eye and the fixation point position of the right eye in the depth image to obtain the left eye middle image and the right eye middle image of the image group; and upsampling the left eye middle image and the right eye middle image of the image group to obtain the sample left eye image and the sample right eye image of the image group.

[0210] In some embodiments, taking the determination of the left eye's gaze point position in the depth image as an example, the server processes the depth image to obtain the first coordinates of the left eye's gaze point position in the depth image. For example, the server processes the depth image based on an eye center localization algorithm to obtain a face image corresponding to the depth image, extracts the ROI region of the left eye from the face image, calculates the position of the left eye's eye center, which is the first coordinate L(ul, vl), for the pixels in the ROI region of the left eye.

[0211] In some embodiments, taking the obtaining of the left-eye center image of the image group as an example, the left-eye image of the image group is processed according to the fixation point position of the left eye in the depth image to obtain the left-eye center image of the image group, including the following steps 308A to 308E:

[0212] In step 308A, the server obtains the first depth value dl corresponding to the first coordinate based on the depth image.

[0213] In step 308B, the server obtains the second coordinates of the left eye's gaze point position in the coordinate system of the depth camera based on the first coordinates and the first depth value dl.

[0214] In some embodiments, before step 308B, the depth camera is calibrated to obtain its intrinsic and extrinsic parameters. The embodiments of this application do not specifically limit the calibration algorithm for the depth camera.

[0215] In some embodiments, the server obtains the second coordinates based on the first coordinates, the first depth value, the intrinsic parameters of the depth camera, and the extrinsic parameters.

[0216] In some embodiments, the intrinsic parameters K and extrinsic parameters D = [R, T] of the depth camera are obtained, where R is the rotation matrix and T is the translation vector. The second coordinate is calculated according to the following formula (1):

[0217]

[0218] In the above formula, M represents the first depth value, (U, V) represents the first coordinate, and K represents the intrinsic parameters of the depth camera.

[0219] For example: M = dl, U = ul, V = vl, then the second coordinate P is calculated according to the above formula (1). l (x l y l , z l ).

[0220] In step 308C, the server obtains the third coordinate of the left eye's gaze point position in the screen center coordinate system based on the second coordinate.

[0221] In some embodiments, the third coordinate is calculated according to the following formula (2):

[0222] P Cl =D*P l (2)

[0223] In the above formula, P Cl The third coordinate is represented by D, which represents the extrinsic parameters of the depth camera, and P represents the extrinsic parameters of the depth camera. l This represents the second coordinate.

[0224] For example, the third coordinate is P Cl (x Cl y Cl , z Cl ).

[0225] In step 308D, the server obtains the updated gaze point position of the left eye based on the third coordinate.

[0226] In some embodiments, the server calculates the angle θ between the line connecting the screen center and the third coordinate and the screen based on the third coordinate, compares θ with multiple fixation angles generated by the prism film of the display, determines the fixation angle corresponding to the updated fixation position, and uses the fixation position corresponding to the fixation angle as the updated fixation position of the left eye.

[0227] In some embodiments, for example, Based on tanθ, we obtain θ. We then compare θ with multiple fixation angles generated by the prism film of the display to determine the fixation angle corresponding to the updated fixation position. The fixation point corresponding to this fixation angle is then used as the updated fixation position for the left eye.

[0228] In step 308E, the server inputs the left eye intermediate image, the depth image, and the updated left eye fixation point position into the image processing tool to obtain the left eye intermediate image.

[0229] In some embodiments, prior to step 1303, the channels of the two color images and the depth image are aligned to obtain a stitched image.

[0230] In some embodiments, the size of the color image and the depth image is 1024*1024*3, and the size of the stitched image is 1024*1024*9.

[0231] In some embodiments, the image processing tool is a neural network model that processes the left eye intermediate image, the depth image, and the updated gaze point position of the left eye to obtain the left eye intermediate image. For example, the image processing tool is ColorNet, and the size of each left eye intermediate image is 1024*1024*3.

[0232] Similarly, the middle image of the right eye can be obtained.

[0233] In some embodiments, the size of the sample left eye image is 2048*2048*3.

[0234] In step 309, a training set is formed by combining the sample left-eye image and the sample right-eye image corresponding to each image group. Based on the training set, the fixation point position prediction model is adjusted.

[0235] In some embodiments, taking a sample left-eye image as an example, the server predicts the gaze point position of the sample left-eye image based on the gaze point position prediction model, obtaining a target left-eye gaze image patch of the sample left-eye image. The target left-eye gaze image patch is then compared with the updated gaze point position of the corresponding left eye in the sample left-eye image to obtain the error of the gaze point position prediction model. In embodiments of this application, the model parameters of the gaze point position prediction model are adjusted online based on the error of the model, making the model more adaptable to the images acquired in real-time by the camera group, thereby predicting a more accurate gaze point position for the real-time acquired images.

[0236] The above embodiments are illustrated using a server as the execution subject, but a terminal can also be used as the execution subject.

[0237] In the embodiments of this application, by determining the label of each sample image, invalid training data is eliminated, and effective training data for a specific task is obtained, thereby shortening the pre-training time of the gaze point location prediction model and improving the accuracy of the gaze point location prediction model.

[0238] Figure 8 This is a flowchart illustrating a pre-training method for a gaze point location prediction model according to an embodiment of this application, as shown below. Figure 8 As shown, this embodiment of the application uses a terminal or server as an example for illustration. The method includes the following steps:

[0239] In step 801, the server acquires multiple second sample images and obtains a target gaze image block for each second sample image based on the gaze point location prediction model. Each second sample image includes multiple image blocks, and the target gaze image block is the image block in the second sample image that includes the most gaze point locations.

[0240] In some embodiments, the second sample image is a 1k image, that is, the size of the second sample image is 1024*1024.

[0241] In some embodiments, the gaze location prediction model divides each second sample image into multiple image blocks, and determines one image block from each of these multiple image blocks as the target gaze image block for that second sample image, thereby identifying the image block that the user is most interested in for each second sample image. For example, the gaze location prediction model divides each second sample image into 100 image blocks, and determines the image block corresponding to the 3rd row and 4th column of the second sample image as the target gaze image block for that second sample image.

[0242] In step 802, the server crops the corresponding second sample image based on each target gaze image block to obtain multiple first sample sub-images. Each first sample sub-image includes the target gaze image block, and the size of the first sample sub-image is smaller than the size of the second sample image.

[0243] In some embodiments, a first sample sub-image is cropped from the corresponding second sample image based on each target gaze image block, in order to retain the image block that the user is most interested in and a portion of the image blocks around it, while the remaining portion is the image block that the user is not interested in and is discarded, thereby reducing the size of the second sample image.

[0244] In step 803, the server generates a training dataset based on multiple of the first sample sub-images.

[0245] In step 804, the server pre-trains the super-resolution model based on the training dataset.

[0246] The above embodiments are illustrated using a server as the execution subject, but a terminal can also be used as the execution subject.

[0247] In the embodiments of this application, a target gaze image patch is determined for each sample image. Based on the target gaze image patch, a first sample sub-image with a size smaller than the corresponding sample image is cropped from each sample image. A training dataset for pre-training the super-resolution model is generated based on multiple first sample sub-images. This allows the super-resolution model to perform super-resolution only on the first sample sub-images during the pre-training process, thereby improving the pre-training speed of the super-resolution model. Furthermore, the pre-training of the super-resolution model is supervised based on a complex pre-defined supervised model, enabling a simple super-resolution model to achieve the super-resolution effect of a complex pre-defined supervised model.

[0248] Figure 8 The illustrated embodiment is a simplified flow of an embodiment of this application. Below, based on... Figure 9 The technical solution of this application will be further described. Figure 9 This is an example flowchart illustrating a pre-training method for a gaze point location prediction model according to an embodiment of this application. In this embodiment, it is illustrated using an application on a terminal or server. The method includes the following steps:

[0249] In step 901, the server predicts the gaze position of each second sample image based on the gaze position prediction model, and obtains multiple labels for each second sample image. Each label includes the coordinates of the gaze image block and the ratio of the gaze image block. The ratio is the ratio of the number of gaze positions in each gaze image block to the total number of gaze positions in the second sample image. The gaze image block is the image block corresponding to the first preset number of positions in the ratio sequence. The ratio sequence is a sequence in which the ratios are arranged from smallest to largest.

[0250] In some embodiments, the gaze position prediction model predicts the gaze position of each second sample image, and then divides each second sample image into m*n gaze image blocks, where m and n are integers greater than or equal to 1. The gaze position prediction model calculates the number of gaze positions in each gaze image block in each second sample image and the total number of gaze positions in each second sample image, and obtains the ratio of the number of gaze positions in each gaze image block in each second sample image to the total number of gaze positions in the corresponding second sample image, thereby obtaining the ratio of each gaze image block in each second sample image.

[0251] In step 902, the server obtains the target ratio for each second sample image, which is the largest ratio in the ratio sequence.

[0252] In step 903, the server uses the gaze image block corresponding to the target ratio as the target gaze image block.

[0253] In step 904, the server takes each target gaze image block as the center, and crops out the first sample sub-image from the corresponding second sample image according to the preset side length, and outputs the remaining part as the second sample sub-image.

[0254] In some embodiments, step 904 includes cropping the first sample sub-image from the corresponding second sample image based on a preset side length, with each target gaze image block as the center, and outputting the remaining part as the second sample sub-image.

[0255] In some embodiments, a preset cropping box (either rectangular or square) is obtained. If the preset cropping box is square, it has a preset side length. The center of the preset cropping box is aligned with the center of the target gaze image block. If the preset cropping box does not exceed the boundary of the sample image, the image portion within the preset cropping box is cropped from the corresponding second sample image to obtain a first sample sub-image of the second sample image. For example, if any second sample image is 1024*1024 in size, after determining the target gaze image block of the second sample image, the center of a preset cropping box of size 512*512 is aligned with the center of the target gaze image block. If the preset cropping box does not exceed the boundary of the second sample image, the image portion within the preset cropping box is cropped from the corresponding second sample image to obtain a first sample sub-image of size 512*512.

[0256] In step 905, the server downsamples each first sample sub-image to obtain multiple downsampled images.

[0257] In some embodiments, pixels in each first sample sub-image are sampled according to a preset rule to obtain multiple downsampled images. Optionally, the preset rule is to extract every other pixel in the first sample sub-image. For example, extracting every other pixel from a first sample sub-image of size 512*512 results in a downsampled image of size 256*256.

[0258] In step 906, the server upsamples each downsampled image to obtain multiple upsampled images included in the training dataset. The upsampled images have the same resolution and size as the corresponding first sample sub-images.

[0259] In some embodiments, taking any downsampled image as an example, the downsampled image is magnified to obtain a magnified image. The magnified image includes multiple locations to be supplemented. Interpolation is performed on each location to be supplemented in the magnified image to obtain an upsampled image. Optionally, interpolation is performed on each location to be supplemented based on a nearest neighbor interpolation algorithm or a bilinear interpolation algorithm. For example, taking a 256*256 downsampled image as an example, the downsampled image is enlarged to obtain a 512*512 enlarged image. The 512*512 enlarged image includes multiple positions to be supplemented. Based on the nearest neighbor interpolation algorithm, each position to be supplemented in the enlarged image is interpolated to obtain a 512*512 upsampled image. The resolution and size of the 512*512 upsampled image are the same as the corresponding first sample sub-image. However, compared with the corresponding first sample sub-image, the pixels obtained by interpolation in the 512*512 upsampled image are invalid pixels, while the pixels in the 256*256 downsampled image are valid pixels. Therefore, the 512*512 upsampled image looks blurry compared to the corresponding first sample sub-image. Thus, the first sample sub-image is a high-definition image, and the 512*512 upsampled image is a low-definition image.

[0260] In step 907, the server pre-trains the super-resolution model based on the training dataset.

[0261] In some embodiments, step 907 includes, during the k-th iteration, inputting the k-th first sample sub-image into the super-resolution model and outputting the k-th first supramolecular image; inputting the k-th first sample sub-image into a preset supervised model and outputting the k-th second supramolecular image; obtaining the k-th first loss value and the k-th second loss value, wherein the k-th first loss value is used to indicate the information difference between the k-th first supramolecular image and the k-th second supramolecular image, and the k-th second loss value is used to indicate the information difference between the k-th second supramolecular image and the k-th first sample sub-image, where k is an integer greater than or equal to 1; determining a loss value and a sum based on the k-th first loss value and the k-th second loss value; and adjusting the model parameters of the super-resolution model based on the loss value and the sum. For example, the super-resolution model is the Unet (image segmentation) model, and the pre-defined supervised model is the diffusionNet (diffusion) model. The Unet model has a simpler model structure than the diffusionNet model. Consequently, the Unet model has a shorter computation time when performing image super-resolution. Therefore, during the pre-training of the super-resolution model based on this training dataset, supervising the pre-training process of the Unet model based on the diffusionNet model not only enables the simple Unet model to achieve the complex effects of the diffusionNet model, but also shortens the computation time.

[0262] In some embodiments, step 907 further includes obtaining a k-th third loss value during the k-th iteration. This k-th third loss value indicates the information difference between the k-th first supramolecular image and the k-th sample sub-image. Determining the sum of losses based on the k-th first loss value and the k-th second loss value includes summing the k-th first loss value, the k-th second loss value, and the k-th third loss value to obtain the sum of losses. Summing the first, second, and third loss values ​​when calculating the sum of losses makes it easier for the super-resolution model to fit compared to calculating only the first and second loss values.

[0263] In step 908, the server acquires multiple image groups of the same target captured by the camera group, each image group including a left eye image, a right eye image, and a depth image;

[0264] In step 909, for each image group, the server processes the left-eye and right-eye images in the image group based on the depth image in the image group to obtain sample left-eye and sample right-eye images.

[0265] In some embodiments, for each image group, the fixation point position of the left eye and the fixation point position of the right eye in the depth image are determined based on the depth image in the image group; based on the fixation point position of the left eye and the fixation point position of the right eye in the depth image, the left eye image and the right eye image of the image group are processed respectively to obtain the left eye middle image and the right eye middle image of the image group; the left eye middle image and the right eye middle image of the image group are upsampled to obtain the sample left eye image and the sample right eye image of the image group.

[0266] Steps 908 and 909 are the same as steps 307 and 308, and will not be repeated here.

[0267] In step 910, the server forms a training set by combining the sample left-eye image and sample right-eye image corresponding to each image group, and adjusts the super-resolution model based on the training set.

[0268] In some embodiments, the server adjusts the super-resolution model based on the training set, including: obtaining a left-eye target gaze image patch for each left-eye image of the sample and a right-eye target gaze image patch for each right-eye image of the sample based on the gaze point location prediction model, wherein each left-eye image of the sample includes multiple left-eye image patches, and the left-eye target gaze image patch is the left-eye image patch with the most gaze point locations in the sample left-eye image; each right-eye image of the sample includes multiple right-eye image patches, and the right-eye target gaze image patch is the right-eye image patch with the most gaze point locations in the sample right-eye image; cropping a first sample left-eye sub-image from the corresponding left-eye image of the sample based on each target left-eye gaze image patch; cropping a first sample right-eye sub-image from the corresponding right-eye image of the sample based on each target right-eye gaze image patch; and adjusting the super-resolution model online based on multiple first sample left-eye sub-images and multiple first sample right-eye images.

[0269] In some embodiments, the server adjusts the super-resolution model based on multiple first sample left-eye images and multiple first sample right-eye images, including: taking the adjustment of the super-resolution model based on multiple first sample left-eye images as an example, the server sequentially downsamples and upsamples the w-th (w is an integer greater than or equal to 1) first sample left-eye image to obtain the w-th upsampled image, and simultaneously inputs the w-th upsampled image into the super-resolution model and the preset supervised model, respectively, and outputs the w-th first left-eye supramolecular image and the w-th second left-eye supramolecular image, respectively. For supramolecular images, the information difference between the w-th first left-eye supramolecular image and the w-th second left-eye supramolecular image is used as the w-th first left-eye loss value. The information difference between the w-th second left-eye supramolecular image and the w-th first sample left-eye sub-image is used as the w-th second left-eye loss value. The information difference between the w-th first left-eye supramolecular image and the w-th first sample left-eye sub-image is used as the w-th third left-eye loss value. The super-resolution model is adjusted online based on the w-th first left-eye loss value, the w-th second left-eye loss value, and the w-th third left-eye loss value.

[0270] Similarly, the super-resolution model is adjusted based on multiple right eye images of the first sample.

[0271] The above embodiments are illustrated using a server as the execution subject, but a terminal can also be used as the execution subject.

[0272] In the embodiments of this application, a target gaze image patch is determined for each sample image. Based on the target gaze image patch, a first sample sub-image with a size smaller than the corresponding sample image is cropped from each sample image. A training dataset for pre-training the super-resolution model is generated based on multiple first sample sub-images. This allows the super-resolution model to perform super-resolution only on the first sample sub-images during the pre-training process, thereby improving the pre-training speed of the super-resolution model. Furthermore, the pre-training of the super-resolution model is supervised based on a complex pre-defined supervised model, enabling a simple super-resolution model to achieve the super-resolution effect of a complex pre-defined supervised model.

[0273] Figure 10 This is a flowchart illustrating an image processing method according to an embodiment of this application, such as... Figure 10 As shown in the embodiments of this application, the method is illustrated using an example of remote communication based on a terminal or server equipped with multiple cameras. The method includes the following steps:

[0274] In step 1001, the server processes the images acquired by the camera group to obtain two initial images.

[0275] In some embodiments, such as Figure 11 As shown, a depth camera is positioned at the center of the top edge of the terminal or server's display. Two color cameras are positioned at the center of the left and right sides of the display. The optical centers of the two color cameras and the depth camera are directed towards the user at a distance D (D > 0) from the screen. The color cameras simulate the human eye. The display has a width of W (W > 0) and a height of H (H > 0). A prism film covers the surface of the display. Adjusting the parameters of this prism film controls the deflection of light emitted from the intersecting pixels to a specified position, thereby generating multiple gaze points within a range. Each gaze point corresponds to a gaze angle. When the prism film is fixed, the gaze angles corresponding to each gaze point are also fixed. For example, as... Figure 12 As shown, Y+1 rays are emitted from the center of the screen Z, where Y is an integer greater than or equal to 1. Each pair of adjacent rays corresponds to a gaze point. Therefore, Figure 11 The diagram illustrates Y fixation points, each capable of displaying a complete image. These images are obtained by cameras capturing images of a stereoscopic object from different positions. When two cameras simulating the human eye are positioned at different fixation points, a stereoscopic object can be captured based on the principle of binocular parallax. The embodiments of this application do not specifically limit the parameters of the prism film.

[0276] In step 1002, the server predicts the gaze location of each initial image based on the gaze location prediction model to obtain the target gaze image block of each initial image. Each initial image includes multiple image blocks, and the target gaze image block is the image block in the initial image that contains the most gaze locations.

[0277] Among them, the target gaze image patch is the image patch that the user is most interested in from the initial image.

[0278] In the embodiments of this application, the target gaze image block determines the new gaze point position of the user in the initial image, causing the position of at least one gaze point of the user in the initial image to change. Based on the principle of binocular parallax, the user sees the reconstructed stereoscopic object, thereby realizing three-dimensional reconstruction.

[0279] In step 1003, the server crops a first initial sub-image from each initial image based on the target gaze image block, and the remaining part is a second initial sub-image. The size of the first initial sub-image is smaller than the size of the initial image.

[0280] The first initial sub-image is an image block in the initial image that the user is interested in, i.e., a valid image block. The second initial sub-image is an image block in the initial image that the user is not interested in, i.e., an invalid image block. The first initial sub-image is cropped out from the initial image to distinguish the image blocks in the initial image that the user is interested in from those that the user is not interested in.

[0281] In step 1004, the server performs super-resolution on each of the first initial sub-images based on the super-resolution model to obtain two first target sub-images.

[0282] In step 1005, the server combines each of the first target sub-images with the corresponding second initial sub-image to obtain two target images.

[0283] In the target image, the first target sub-image is a high-definition image, and the second initial sub-image is a low-definition image. Since the user is only interested in the first initial sub-image in the initial image, only the first initial sub-image is super-resolutioned to transform it from a low-definition image into a high-definition image. The second initial sub-image, which is not of interest to the user, is not super-resolutioned, so that the second initial sub-image remains a low-definition image and does not affect the user's viewing experience.

[0284] In some embodiments, the server stitches the edges of the first target sub-image with the edges of the corresponding second initial sub-image to obtain the target image.

[0285] The above embodiments are illustrated using a server as the execution subject, but a terminal can also be used as the execution subject.

[0286] In the embodiments of this application, the image patch of most interest to the user is determined by a gaze point location prediction model. Based on the image patch of most interest to the user, the image patch of interest to the user is cropped from the image, and the remaining part is the image patch of no interest to the user. The image patch of interest to the user in the image is super-resolution performed by a super-resolution model to obtain a supramolecular image. The supramolecular image is combined with the remaining part of the image to obtain the target image. Without affecting the user's viewing experience, super-resolution is only performed on the image patch of interest to the user in the image, which reduces the consumption of graphics card resources and thus improves the real-time performance of image display.

[0287] Figure 10 The illustrated embodiment is a simplified flow of an embodiment of this application. Below, based on... Figure 13 The technical solution of this application will be further described. Figure 13 This is an example flowchart illustrating an image processing method according to an embodiment of this application. In this embodiment, it is illustrated using an application on a terminal or server equipped with multiple cameras, and remote communication based on the terminal or server as an example. The method includes the following steps:

[0288] In step 1301, the server acquires multiple image groups captured by the camera group of the user who is conducting remote communication. Each image group includes a left-eye image, a right-eye image, and a depth image.

[0289] In some embodiments, the size of both the left-eye image and the right-eye image is 1024*1024*3, where 1024*1024 is the size of the left-eye image, the right-eye image, and the depth image, and 3 is the number of channels.

[0290] In step 1302, for each image group, the left-eye image and the right-eye image in the image group are processed according to the depth image in the image group to obtain two initial images, that is, two new fixation point position images.

[0291] In some embodiments, taking the processing of the left-eye image in the image group to obtain the initial image corresponding to the left-eye image as an example, step 1302 includes the following steps 1302A to 1302E:

[0292] In step 1302A, the server processes the depth image to obtain the first coordinates of the left eye's gaze point position in the depth image.

[0293] In step 1302B, the server obtains the first depth value dl corresponding to the first coordinate based on the depth image.

[0294] In step 1302C, the server obtains the second coordinates of the left eye's gaze point position in the coordinate system of the depth camera based on the first coordinates and the first depth value dl.

[0295] In step 1302D, the server obtains the third coordinate of the left eye's gaze point position in the screen center coordinate system based on the second coordinate.

[0296] In step 1302E, the server obtains the new fixation point position of the left eye based on the third coordinate.

[0297] Similarly, the initial image corresponding to the right eye image can be obtained.

[0298] Steps 1302A to 1302E are the same as steps 308A to 308E, and will not be described again.

[0299] In step 1303, the server inputs the left eye image, the right eye image, the depth image, the new fixation point position of the left eye, and the new fixation point position of the right eye into the image processing tool, and outputs two initial images, namely the new fixation point position image of the left eye and the new fixation point position image of the right eye.

[0300] In step 1304, the server upsamples the image at each new gaze point location to obtain two upsampled images.

[0301] In step 1305, the server predicts the gaze location of each upsampled image based on the gaze location prediction model to obtain the target gaze image block of each upsampled image.

[0302] In step 1306, the server crops a first upsampled sub-image from each upsampled image based on the target gaze image block, and the remaining part is a second upsampled sub-image. The size of the first upsampled sub-image is smaller than the size of the upsampled image.

[0303] Steps 1305 to 1306 are the same as steps 901 to 904, and will not be described again.

[0304] In step 1307, the server performs super-resolution on each first upsampled sub-image based on the super-resolution model to obtain two first target sub-images.

[0305] In some embodiments, the pre-training process of the super-resolution model includes: predicting the gaze position of multiple second sample images based on the gaze position prediction model to obtain a gaze image patch for each second sample image; cropping a first sample sub-image from each second sample image based on each gaze image patch; and pre-training the super-resolution model based on the multiple first sample sub-images and each first sample sub-image.

[0306] In some embodiments, the adjustment process of the super-resolution model includes: acquiring multiple image groups acquired by a camera group of the same target, each image group including a left-eye image, a right-eye image, and a depth image; for each image group, processing the left-eye image and the right-eye image in the image group according to the depth image in the image group to obtain a sample left-eye image and a sample right-eye image; forming a training set with the sample left-eye image and the sample right-eye image corresponding to each image group, and adjusting the super-resolution model based on the training set.

[0307] In step 1308, the server combines each first target sub-image with the corresponding second upsampled sub-image to obtain two target images.

[0308] In some embodiments, the server concatenates the edges of each first target sub-image with the edges of the corresponding second upsampled sub-image to obtain two target images.

[0309] In some embodiments, since the gaze point positions of the two target images have changed relative to the depth image, the two target images are stitched together according to the principle of binocular parallax to obtain a reconstructed 3D image, thereby achieving 3D reconstruction. Furthermore, since graphics card resources are saved during the image super-resolution process, the 3D reconstruction speed is also improved.

[0310] The above embodiments are illustrated using a server as the execution subject, but a terminal can also be used as the execution subject.

[0311] In the embodiments of this application, the image patch of most interest to the user is determined by a gaze point location prediction model. Based on the image patch of most interest to the user, the image patch of interest to the user is cropped from the image, and the remaining part is the image patch of no interest to the user. The image patch of interest to the user in the image is super-resolution performed by a super-resolution model to obtain a supramolecular image. The supramolecular image is combined with the remaining part of the image to obtain the target image. Without affecting the user's viewing experience, super-resolution is only performed on the image patch of interest to the user in the image, which reduces the consumption of graphics card resources and thus improves the real-time performance of image display.

[0312] Figure 14 This is a schematic diagram of a gaze point location prediction model training device 1400 provided according to an embodiment of this application. The device includes:

[0313] The first acquisition module 1401 is used to acquire multiple first sample images and determine a first ratio of each image block in each first sample image. The first ratio is the ratio of the number of fixation point positions in each image block to the total number of fixation point positions in the first sample image.

[0314] The sorting module 1402 is used to sort the multiple first ratios of each first sample image from largest to smallest to obtain a ratio sequence;

[0315] The second acquisition module 1403 is used to acquire multiple second ratios for each of the first sample images from the ratio sequence, wherein the multiple second ratios are located in the first preset number of positions in the ratio sequence;

[0316] The third acquisition module 1404 is used to acquire multiple first labels for each of the first sample images. The first label includes the coordinates of the first gaze image block and the second ratio of the first gaze image block. The first gaze image block is used to indicate the image block corresponding to the second ratio.

[0317] The pre-training module 1405 is used to pre-train a gaze point location prediction model based on multiple first sample images and multiple first labels for each first sample image.

[0318] In one possible implementation, the pre-training module 1405 includes:

[0319] An iterative unit is used to predict the gaze position of the i-th first sample image through the gaze position prediction model during the i-th iteration, and obtain multiple second labels for the i-th first sample image. The second labels include the coordinates of multiple second gaze image blocks and the third ratio of the second gaze image block. The third ratio is used to indicate the number of gaze positions in the second gaze image block, where i is an integer greater than or equal to 1.

[0320] An acquisition unit is used to acquire the i-th error value, which is used to indicate the ratio difference between each of the first gaze image blocks and the second gaze image block with the same coordinates;

[0321] The judgment unit is used to stop pre-training when the i-th iteration process meets the preset conditions.

[0322] In one possible implementation, the acquiring unit includes:

[0323] Determine sub-units to respectively determine the second label that has the same coordinates as each of the first labels;

[0324] The sub-unit is configured to acquire multiple first differences, which are used to indicate the difference between the second ratio and the third ratio corresponding to the same coordinate, and to acquire multiple second differences, which are used to indicate the second ratio that does not correspond to the third ratio at the same coordinate.

[0325] The calculation subunit is used to sum multiple such differences to obtain the i-th error value.

[0326] In one possible implementation, the device further includes:

[0327] The acquisition module is used to acquire multiple image groups of the same target acquired by the camera group. Each image group includes a left-eye image, a right-eye image, and a depth image.

[0328] The processing module is used to process the left-eye and right-eye images in each image group based on the depth image in the image group, so as to obtain sample left-eye and sample right-eye images.

[0329] The adjustment module is used to form a training set by combining the sample left-eye image and sample right-eye image corresponding to each image group, and to adjust the gaze point position prediction model based on the training set.

[0330] In one possible implementation, the processing module includes:

[0331] The determining unit is used to determine, for each image group, the fixation point position of the left eye and the fixation point position of the right eye in the depth image of the image group.

[0332] The processing unit is used to process the left-eye image and the right-eye image of the image group according to the fixation point position of the left eye and the fixation point position of the right eye in the depth image, respectively, to obtain the left-eye middle image and the right-eye middle image of the image group.

[0333] The upsampling unit is used to upsample the left-eye middle image and the right-eye middle image of the image group to obtain the sample left-eye image and sample right-eye image of the image group.

[0334] It should be noted that the gaze point location prediction model pre-training device provided in the above embodiments is only illustrated by the division of the above functional modules when performing the corresponding steps. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the gaze point location prediction model pre-training device and the gaze point location prediction model pre-training method embodiment provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0335] In the embodiments of this application, by determining the label of each sample image, invalid training data is eliminated, and effective training data for a specific task is obtained, thereby shortening the pre-training time of the gaze point location prediction model and improving the accuracy of the gaze point location prediction model.

[0336] Figure 15This is a schematic diagram of a super-resolution model pre-training device 1500 according to an embodiment of this application. The device includes:

[0337] The acquisition module 1501 is used to acquire multiple second sample images and obtain a target gaze image block for each second sample image based on a gaze point location prediction model. Each second sample image includes multiple image blocks, and the target gaze image block is the image block in the second sample image that includes the most gaze point locations.

[0338] The cropping module 1502 is used to crop the corresponding second sample image based on each target gaze image block to obtain a plurality of first sample sub-images, each of the first sample sub-images including the target gaze image block, and the size of the first sample sub-image is smaller than the size of the second sample image;

[0339] Generation module 1503 is used to generate a training dataset based on multiple of the first sample sub-images;

[0340] The pre-training module 1504 is used to pre-train the super-resolution model based on the training dataset.

[0341] In one possible implementation, the acquisition module 1501 includes:

[0342] The prediction unit is used to predict the gaze position of each second sample image based on the gaze position prediction model, and obtain multiple labels for each second sample image. Each label includes the coordinates of the gaze image block and the ratio of the gaze image block. The ratio is the ratio of the number of gaze positions in each gaze image block to the total number of gaze positions in the second sample image. The gaze image block is the image block corresponding to the first preset number of positions in the ratio sequence. The ratio sequence is a sequence in which the ratios are arranged from smallest to largest.

[0343] An acquisition unit is used to acquire the target ratio of each of the second sample images, wherein the target ratio is the largest ratio in the ratio sequence;

[0344] The first determining unit is used to determine the gaze image block corresponding to the target ratio as the target gaze image block.

[0345] In one possible implementation, the trimming module 1502 includes:

[0346] The first cropping unit is used to crop the first sample sub-image from the corresponding second sample image based on a preset side length, with each target gaze image block as the center, and output the remaining part as the second sample sub-image.

[0347] In one possible implementation, the generation module 1503 includes:

[0348] The downsampling unit is used to downsample each of the first sample sub-images to obtain multiple downsampled images;

[0349] The first upsampling unit is used to upsample each of the downsampled images to obtain multiple upsampled images included in the training dataset. The resolution and size of the upsampled images are the same as those of the corresponding first sample sub-images.

[0350] In one possible implementation, the pre-training module 1504 includes:

[0351] The first iteration unit is used to, during the k-th iteration, input the k-th first sample sub-image into the super-resolution model and output the k-th first supramolecular image; input the k-th first sample sub-image into a preset supervised model and output the k-th second supramolecular image; and obtain the k-th first loss value and the k-th second loss value. The k-th first loss value is used to indicate the information difference between the k-th first supramolecular image and the k-th second supramolecular image, and the k-th second loss value is used to indicate the information difference between the k-th second supramolecular image and the k-th first sample sub-image, where k is an integer greater than or equal to 1.

[0352] The first calculation unit is used to determine the loss value and the value based on the kth first loss value and the kth second loss value;

[0353] The adjustment unit is used to adjust the model parameters of the super-resolution model based on the loss value and the value.

[0354] In one possible implementation, the pre-training module 1504 further includes:

[0355] The second iteration unit is used to obtain the k-th third loss value during the k-th iteration. The k-th third loss value is used to indicate the information difference between the k-th first supramolecular image and the k-th first sample sub-image.

[0356] The first computing unit includes:

[0357] The calculation subunit is used to sum the k-th first loss value, the k-th second loss value, and the k-th third loss value to obtain the sum of the loss values.

[0358] In one possible implementation, the device further includes:

[0359] The acquisition module is used to acquire multiple image groups of the same target acquired by the camera group. Each image group includes a left-eye image, a right-eye image, and a depth image.

[0360] The processing module is used to process the left-eye and right-eye images in each image group based on the depth image in the image group, so as to obtain sample left-eye and sample right-eye images.

[0361] The adjustment module is used to form a training set by combining the left-eye image and the right-eye image of each sample in the image group, and to adjust the super-resolution model based on the training set.

[0362] In one possible implementation, the processing module includes:

[0363] The second determining unit is used to determine, for each image group, the fixation point position of the left eye and the fixation point position of the right eye in the depth image of the image group.

[0364] The processing unit is used to process the left-eye image and the right-eye image of the image group according to the fixation point position of the left eye and the fixation point position of the right eye in the depth image, respectively, to obtain the left-eye middle image and the right-eye middle image of the image group.

[0365] The second upsampling unit is used to upsample the left-eye middle image and the right-eye middle image of the image group to obtain the sample left-eye image and sample right-eye image of the image group.

[0366] In one possible implementation, the adjustment module includes:

[0367] The second prediction unit is used to obtain the left-eye target gaze image block of each left-eye image of the sample and the right-eye target gaze image block of each right-eye image of the sample based on the gaze point location prediction model. Each left-eye image of the sample includes multiple left-eye image blocks, and the left-eye target gaze image block is the left-eye image block with the most gaze point locations in the left-eye image of the sample. Each right-eye image of the sample includes multiple right-eye image blocks, and the right-eye target gaze image block is the right-eye image block with the most gaze point locations in the right-eye image of the sample.

[0368] The second cropping unit is used to crop out a first sample left eye sub-image from the corresponding sample left eye image for each left eye gaze image block of the target; and to crop out a first sample right eye sub-image from the corresponding sample right eye image for each right eye gaze image block of the target.

[0369] The adjustment unit is used to adjust the super-resolution model based on multiple left-eye images and multiple right-eye images of the first sample.

[0370] It should be noted that the super-resolution model pre-training device provided in the above embodiments is only illustrated by the division of the above functional modules when performing the corresponding steps. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the super-resolution model pre-training device and the super-resolution model pre-training method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0371] In the embodiments of this application, a target gaze image patch is determined for each sample image. Based on the target gaze image patch, a first sample sub-image with a size smaller than the corresponding sample image is cropped from each sample image. A training dataset for training the super-resolution model is generated based on multiple first sample sub-images. This ensures that during the pre-training process of the super-resolution model, the super-resolution model only needs to perform super-resolution on the first sample sub-images, thereby improving the pre-training speed of the super-resolution model. Furthermore, the pre-training of the super-resolution model is supervised based on a complex pre-defined supervised model, enabling a simple super-resolution model to achieve the super-resolution effect of a complex pre-defined supervised model.

[0372] Figure 16 This is a schematic diagram of the structure of an image processing apparatus 1600 according to an embodiment of this application. The apparatus includes:

[0373] The processing module 1601 is used to process the images acquired by the camera group to obtain two initial images;

[0374] The prediction module 1602 is used to predict the gaze position of each initial image based on the gaze position prediction model to obtain the target gaze image block of each initial image. Each initial image includes multiple image blocks, and the target gaze image block is the image block in the initial image that contains the most gaze positions.

[0375] The cropping module 1603 is used to crop a first initial sub-image from each of the initial images based on the target gaze image block, with the remaining part being a second initial sub-image, the size of which is smaller than the size of the initial image;

[0376] The super-resolution module 1604 is used to perform super-resolution on each of the first initial sub-images based on the super-resolution model to obtain two first target sub-images;

[0377] The combination module 1605 is used to combine each of the first target sub-images with the corresponding second initial sub-image to obtain two target images.

[0378] It should be noted that the image processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when performing the corresponding steps. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing apparatus and image processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0379] In the embodiments of this application, the image patch of most interest to the user is determined by a gaze point location prediction model. Based on the image patch of most interest to the user, the image patch of interest to the user is cropped from the image, and the remaining part is the image patch of no interest to the user. The image patch of interest to the user in the image is super-resolution performed by a super-resolution model to obtain a supramolecular image. The supramolecular image is combined with the remaining part of the image to obtain the target image. Without affecting the user's viewing experience, super-resolution is only performed on the image patch of interest to the user in the image, which reduces the consumption of graphics card resources and thus improves the real-time performance of image display.

[0380] Embodiments of this application also provide a computer device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the above-described gaze position prediction model pre-training method, the above-described super-resolution model pre-training method, or the above-described image processing method.

[0381] Taking computer devices as terminals as an example, Figure 17 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. See also... Figure 17 Terminal 1700 can be: a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. Terminal 1700 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0382] Typically, terminal 1700 includes a processor 1701 and a memory 1702.

[0383] Processor 1701 may include one or more processing cores, such as a 4-core processor, a 17-core processor, etc. Processor 1701 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1701 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1701 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1701 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0384] The memory 1702 may include one or more computer-readable storage media, which may be non-transitory. The memory 1702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1702 are used to store at least one program code, which is executed by the processor 1701 to implement the terminal execution process in the gaze position prediction model pre-training method, super-resolution model pre-training method, or image processing method provided in the method embodiments of this disclosure.

[0385] In some embodiments, the terminal 1700 may also optionally include a peripheral device interface 1703 and at least one peripheral device. The processor 1701, memory 1702, and peripheral device interface 1703 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1703 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1704, a display screen 1705, a camera assembly 1706, an audio circuit 1707, and a power supply 1708.

[0386] Peripheral device interface 1703 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1701 and memory 1702. In some embodiments, processor 1701, memory 1702 and peripheral device interface 1703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1701, memory 1702 and peripheral device interface 1703 can be implemented on separate chips or circuit boards, which is not limited in this disclosure.

[0387] The radio frequency (RF) circuit 1704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1704 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. In some embodiments, the RF circuit 1704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1704 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1704 may also include circuitry related to NFC (Near Field Communication), which is not limited herein.

[0388] Display screen 1705 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1705 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1701 for processing. In this case, display screen 1705 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1705, disposed on the front panel of terminal 1700; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1700 or in a folded design; in still other embodiments, display screen 1705 may be a flexible display screen, disposed on a curved or folded surface of terminal 1700. Furthermore, display screen 1705 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1705 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0389] The camera assembly 1706 is used to acquire images or videos. In some embodiments, the camera assembly 1706 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1706 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0390] The audio circuit 1707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1701 for processing, or input to the radio frequency circuit 1704 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1700. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1701 or the radio frequency circuit 1704 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1707 may also include a headphone jack.

[0391] Power supply 1708 is used to power the various components in terminal 1700. Power supply 1708 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1708 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0392] In some embodiments, the terminal 1700 further includes one or more sensors 1709. The one or more sensors 1709 include, but are not limited to: an accelerometer 1710, a gyroscope 1711, a pressure sensor 1712, an optical sensor 1713, and a proximity sensor 1714.

[0393] Accelerometer 1710 can detect the magnitude of acceleration along the three axes of a coordinate system established by terminal 1700. For example, accelerometer 1710 can be used to detect the components of gravitational acceleration along the three axes. Processor 1701 can control display screen 1705 to display the user page in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1710. Accelerometer 1710 can also be used for collecting game or user motion data.

[0394] The gyroscope sensor 1711 can detect the orientation and rotation angle of the terminal 1700. The gyroscope sensor 1711, in conjunction with the accelerometer sensor 1710, can collect the user's three-dimensional movements of the terminal 1700. Based on the data collected by the gyroscope sensor 1711, the processor 1701 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0395] The pressure sensor 1712 can be disposed on the side bezel of the terminal 1700 and / or on the lower layer of the display screen 1705. When the pressure sensor 1712 is disposed on the side bezel of the terminal 1700, it can detect the user's grip signal on the terminal 1700, and the processor 1701 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1712. When the pressure sensor 1712 is disposed on the lower layer of the display screen 1705, the processor 1701 can control the operable controls on the UI page based on the user's pressure operation on the display screen 1705. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0396] Optical sensor 1713 is used to collect ambient light intensity. In one embodiment, processor 1701 can control the display brightness of display screen 1705 based on the ambient light intensity collected by optical sensor 1713. Specifically, when the ambient light intensity is high, the display brightness of display screen 1705 is increased; when the ambient light intensity is low, the display brightness of display screen 1705 is decreased. In another embodiment, processor 1701 can also dynamically adjust the shooting parameters of camera assembly 1706 based on the ambient light intensity collected by optical sensor 1713.

[0397] The proximity sensor 1714, also known as a distance sensor, is typically located on the front panel of the terminal 1700. The proximity sensor 1714 is used to detect the distance between the user and the front of the terminal 1700. In one embodiment, when the proximity sensor 1714 detects that the distance between the user and the front of the terminal 1700 is gradually decreasing, the processor 1701 controls the display screen 1705 to switch from a screen-on state to a screen-off state; when the proximity sensor 1714 detects that the distance between the user and the front of the terminal 1700 is gradually increasing, the processor 1701 controls the display screen 1705 to switch from a screen-off state to a screen-on state.

[0398] Those skilled in the art will understand that Figure 17 The structure shown does not constitute a limitation on terminal 1700 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0399] Taking computer equipment as a server as an example, Figure 18This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1800 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1801 and one or more memories 1802. The one or more memories 1802 store at least one computer program, which is loaded and executed by the one or more processors 1801 to implement the aforementioned gaze point location prediction model pre-training method, the aforementioned super-resolution model pre-training method, or the aforementioned image processing method. Of course, the server 1800 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1800 may also include other components for implementing device functions, which will not be elaborated here.

[0400] Embodiments of this application also provide a computer-readable storage medium, which includes a stored computer program. When the computer program runs, it controls the device containing the computer-readable storage medium to execute the aforementioned gaze position prediction model pre-training method, the aforementioned super-resolution model pre-training method, or the aforementioned image processing method. Optionally, the computer-readable storage medium may be read-only memory (ROM), random access memory (RAM), compact-disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0401] Embodiments of this application also provide a computer program product, which, when executed by a processor, implements the above-described gaze position prediction model pre-training method, super-resolution model pre-training method, or image processing method.

[0402] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0403] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A pre-training method for a gaze point location prediction model, characterized in that, include: Acquire multiple first sample images, and determine a first ratio for each image patch in each first sample image. The first ratio is the ratio of the number of fixation point locations in each image patch to the total number of fixation point locations in the first sample image. For each first sample image, sort the multiple first ratios from largest to smallest to obtain a ratio sequence; Multiple second ratios are obtained for each first sample image from the ratio sequence, wherein the multiple second ratios are located in the first preset number of positions of the ratio sequence; Obtain multiple first labels for each first sample image, the first label including the coordinates of a first gaze image block and a second ratio of the first gaze image block, the first gaze image block being used to indicate the image block corresponding to the second ratio; In the i-th iteration, the gaze position prediction model is used to predict the gaze position of the i-th first sample image to obtain multiple second labels for the i-th first sample image. The second labels include the coordinates of multiple second gaze image blocks and the third ratio of the second gaze image blocks. The third ratio is used to indicate the number of gaze positions in the second gaze image blocks, where i is an integer greater than or equal to 1. Obtain the i-th error value, which is used to indicate the ratio difference between each first gaze image block and a second gaze image block with the same coordinates; When the i-th iteration process meets the preset conditions, the pre-training stops, and the trained gaze point position prediction model is obtained.

2. The method according to claim 1, characterized in that, Obtaining the i-th error value includes: Determine the second label that has the same coordinates as each of the first labels; A plurality of first differences are obtained, the first differences being used to indicate the difference between the second ratio and the third ratio corresponding to the same coordinate; and a plurality of second differences are obtained, the second differences being used to indicate the second ratio that does not correspond to the third ratio. The i-th error value is obtained by summing the multiple differences.

3. The method according to claim 1, characterized in that, When the i-th iteration process meets the preset conditions, the pre-training stops, and the trained gaze point position prediction model is obtained, the method further includes: Acquire multiple image groups of the same target captured by a camera group, each image group including a left-eye image, a right-eye image, and a depth image; For each image group, the left-eye and right-eye images in the image group are processed according to the depth image in the image group to obtain sample left-eye and sample right-eye images; The left-eye image and right-eye image of each image group are combined to form a training set, and the fixation point position prediction model is adjusted based on the training set.

4. The method according to claim 3, characterized in that, The step of processing the left-eye and right-eye images in each image group based on the depth image in the image group to obtain sample left-eye and sample right-eye images includes: For each image group, based on the depth image in the image group, determine the fixation point position of the left eye and the fixation point position of the right eye in the depth image; Based on the fixation point positions of the left and right eyes in the depth image, the left and right eye images of the image group are processed respectively to obtain the left eye middle image and the right eye middle image of the image group. Upsample the left-eye center image and the right-eye center image of the image group to obtain the sample left-eye image and the sample right-eye image of the image group.

5. A pre-training method for a super-resolution model, characterized in that, include: Multiple second sample images are acquired, and a target gaze image patch is obtained for each second sample image based on the gaze location prediction model. Each second sample image includes multiple image patches, and the target gaze image patch is the image patch in the second sample image that includes the most gaze locations. The gaze location prediction model is trained based on the pre-training method of the gaze location prediction model according to any one of claims 1 to 4. Based on each target gaze image block, the corresponding second sample image is cropped to obtain multiple first sample sub-images. Each first sample sub-image includes the target gaze image block, and the size of the first sample sub-image is smaller than the size of the second sample image. A training dataset is generated based on multiple first sample sub-images; The super-resolution model is pre-trained based on the training dataset.

6. The method according to claim 5, characterized in that, The gaze point location prediction model yields the target gaze image patch for each of the second sample images, including: Based on the gaze point location prediction model, gaze point location is predicted for each second sample image to obtain multiple labels for each second sample image. Each label includes the coordinates of the gaze image patch and the ratio of the gaze image patch. The ratio is the ratio of the number of gaze point locations in each gaze image patch to the total number of gaze point locations in the second sample image. The gaze image patch is the image patch corresponding to the first preset number of positions in the ratio sequence. The ratio sequence is a sequence in which the ratios are arranged from smallest to largest. Obtain the target ratio for each of the second sample images, where the target ratio is the largest ratio in the ratio sequence; The gaze image block corresponding to the target ratio is taken as the target gaze image block.

7. The method according to claim 5, characterized in that, The second sample image is cropped based on each of the target gaze image blocks to obtain multiple first sample sub-images, including: Centered on each of the target gaze image blocks, the first sample sub-image is cropped from the corresponding second sample image according to a preset side length, and the remaining part is output as the second sample sub-image.

8. The method according to claim 5, characterized in that, The generation of the training dataset based on multiple first sample sub-images includes: Each of the first sample sub-images is downsampled to obtain multiple downsampled images; Each downsampled image is upsampled to obtain multiple upsampled images included in the training dataset. The upsampled images have the same resolution and size as the corresponding first sample sub-image.

9. The method according to claim 5, characterized in that, The pre-training of the super-resolution model based on the training dataset includes: In the k-th iteration, the k-th first sample sub-image is input into the super-resolution model, outputting the k-th first supramolecular image. The k-th first sample sub-image is then input into a preset supervised model, outputting the k-th second supramolecular image. The k-th first loss value and the k-th second loss value are obtained. The k-th first loss value indicates the information difference between the k-th first supramolecular image and the k-th second supramolecular image, and the k-th second loss value indicates the information difference between the k-th second supramolecular image and the k-th first sample sub-image. k is an integer greater than or equal to 1. Based on the kth first loss value and the kth second loss value, determine the loss value and the value; Based on the loss value and the value, the model parameters of the super-resolution model are adjusted.

10. The method according to claim 9, characterized in that, The pre-training of the super-resolution model based on the training dataset also includes: During the k-th iteration, the k-th third loss value is obtained. This k-th third loss value is used to indicate the information difference between the k-th first supramolecular image and the k-th first sample sub-image. The determination of the loss value and the value based on the kth first loss value and the kth second loss value includes: The sum of the k-th first loss value, the k-th second loss value, and the k-th third loss value is calculated to obtain the sum of the loss values.

11. The method according to claim 5, characterized in that, After pre-training the super-resolution model based on the training dataset, the method further includes: Acquire multiple image groups of the same target captured by a camera group, each image group including a left-eye image, a right-eye image, and a depth image; For each image group, the left-eye and right-eye images in the image group are processed according to the depth image in the image group to obtain sample left-eye and sample right-eye images; The left-eye image and right-eye image of each image group are combined to form a training set, and the super-resolution model is adjusted based on the training set.

12. The method according to claim 11, characterized in that, The step of processing the left-eye and right-eye images in each image group based on the depth image in the image group to obtain sample left-eye and sample right-eye images includes: For each image group, based on the depth image in the image group, determine the fixation point position of the left eye and the fixation point position of the right eye in the depth image; Based on the fixation point positions of the left and right eyes in the depth image, the left and right eye images of the image group are processed respectively to obtain the left eye middle image and the right eye middle image of the image group. Upsample the left-eye center image and the right-eye center image of the image group to obtain the sample left-eye image and the sample right-eye image of the image group.

13. The method according to claim 11, characterized in that, The adjustment of the super-resolution model based on the training set includes: Based on the fixation point location prediction model, the left-eye target fixation image block of each sample left-eye image and the right-eye target fixation image block of each sample right-eye image in the training set are obtained. Each sample left-eye image includes multiple left-eye image blocks, and the left-eye target fixation image block is the left-eye image block with the most fixation point locations in the sample left-eye image. Each sample right-eye image includes multiple right-eye image blocks, and the right-eye target fixation image block is the right-eye image block with the most fixation point locations in the sample right-eye image. Based on each target left-eye gaze image block, a first sample left-eye sub-image is cropped from the corresponding sample left-eye image; based on each target right-eye gaze image block, a first sample right-eye sub-image is cropped from the corresponding sample right-eye image. The super-resolution model is adjusted based on multiple left-eye images of the first sample and multiple right-eye images of the first sample.

14. An image processing method, characterized in that, include: The images acquired by the camera group are processed to obtain two initial images; Based on the gaze location prediction model, gaze location prediction is performed on each of the initial images to obtain the target gaze image block of each initial image. Each initial image includes multiple image blocks, and the target gaze image block is the image block in the initial image that contains the most gaze locations. The gaze location prediction model is trained based on the pre-training method of the gaze location prediction model according to any one of claims 1 to 4. Based on the target gaze image block, a first initial sub-image is cropped from each of the initial images, and the remaining part is the second initial sub-image. The size of the first initial sub-image is smaller than the size of the initial image. Each of the first initial sub-images is super-splittered based on the super-resolution model to obtain two first target sub-images. The super-resolution model is trained based on the pre-training method of the super-resolution model according to any one of claims 5 to 13. Each of the first target sub-images is combined with the corresponding second initial sub-image to obtain two target images.

15. A pre-training device for a gaze point location prediction model, characterized in that, include: The first acquisition module is used to acquire multiple first sample images and determine a first ratio of each image block in each first sample image. The first ratio is the ratio of the number of fixation point positions in each image block to the total number of fixation point positions in the first sample image. The sorting module is used to sort the multiple first ratios of each first sample image from largest to smallest to obtain a ratio sequence; The second acquisition module is used to acquire multiple second ratios for each first sample image from the ratio sequence, wherein the multiple second ratios are located in the first preset number of positions in the ratio sequence; The third acquisition module is used to acquire multiple first labels for each first sample image. The first label includes the coordinates of a first gaze image block and a second ratio of the first gaze image block. The first gaze image block is used to indicate the image block corresponding to the second ratio. The pre-training module is used to predict the gaze position of the i-th first sample image through the gaze position prediction model during the i-th iteration, and obtain multiple second labels for the i-th first sample image. The second labels include the coordinates of multiple second gaze image patches and a third ratio of the second gaze image patches. The third ratio is used to indicate the number of gaze positions in the second gaze image patches, where i is an integer greater than or equal to 1. The module also obtains the i-th error value, which is used to indicate the ratio difference between each first gaze image patch and a second gaze image patch with the same coordinates. When the i-th iteration process meets the preset conditions, the pre-training stops, and the trained gaze point position prediction model is obtained.

16. A super-resolution model pre-training device, characterized in that, include: The acquisition module is used to acquire multiple second sample images and obtain a target gaze image block for each second sample image based on the gaze point location prediction model. Each second sample image includes multiple image blocks, and the target gaze image block is the image block in the second sample image that includes the most gaze points. The gaze point location prediction model is trained based on the pre-training device of the gaze point location prediction model according to claim 15. The cropping module is used to crop the corresponding second sample image based on each target gaze image block to obtain a plurality of first sample sub-images, each first sample sub-image including the target gaze image block, and the size of the first sample sub-image is smaller than the size of the second sample image; The generation module is used to generate a training dataset based on multiple first sample sub-images; The pre-training module is used to pre-train the super-resolution model based on the training dataset.

17. An image processing apparatus, characterized in that, include: The processing module is used to process the images acquired by the camera group to obtain two initial images; The prediction module is used to predict the gaze position of each initial image based on the gaze position prediction model to obtain the target gaze image block of each initial image. Each initial image includes multiple image blocks, and the target gaze image block is the image block in the initial image that contains the most gaze positions. The gaze position prediction model is trained based on the pre-training device of the gaze position prediction model according to claim 15. A cropping module is used to crop a first initial sub-image from each of the initial images based on the target gaze image block, with the remaining portion being a second initial sub-image, wherein the size of the first initial sub-image is smaller than the size of the initial image; The super-resolution module is used to perform super-resolution on each of the first initial sub-images based on the super-resolution model to obtain two first target sub-images, wherein the super-resolution model is trained based on the pre-training device of the super-resolution model as described in claim 16; The combination module is used to combine each of the first target sub-images with the corresponding second initial sub-image to obtain two target images.

18. A computer device, characterized in that, The computer device includes a processor and a memory, the memory being used to store at least one program, the at least one program being loaded by the processor and executed as a pre-training method for a gaze point location prediction model as claimed in any one of claims 1 to 4, or a pre-training method for a super-resolution model as claimed in any one of claims 5 to 13, or an image processing method as claimed in claim 14.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the gaze point location prediction model pre-training method as described in any one of claims 1 to 4, the super-resolution model pre-training method as described in any one of claims 5 to 13, or the image processing method as described in claim 14.

20. A computer program product, characterized in that, The computer program product includes at least one program segment loaded and executed by a processor to implement the gaze point location prediction model pre-training method as described in any one of claims 1 to 4, the super-resolution model pre-training method as described in any one of claims 5 to 13, or the image processing method as described in claim 14.

Images