Image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment and a storage medium. The image processing method comprises the following steps: acquiring eye movement data of a target object watching a display image of the electronic equipment; based on the eye movement data, correcting the display image to obtain a target image; and displaying based on the target image. According to the embodiment of the invention, self-adaptive correction of the display image of the electronic equipment can be realized according to different eye movement data of the target object, so that the display effect of the electronic equipment and the use experience of a user are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of display technology, and in particular to an image processing method and device, an electronic device, and a storage medium. Background Art

[0002] With the gradual development of hardware technology, the application of augmented reality (AR) technology is becoming more and more widespread. Among them, electronic devices such as AR devices refer to devices that can see the real world and virtual information at the same time, that is, superimpose virtual information in the form of "overlay" in front of the line of sight, so as to achieve the effect of integrating virtual information and real scenes.

[0003] Currently, AR devices such as head-mounted devices such as AR glasses usually produce problems such as distortion, dispersion, and shadow on the display screen.

[0004] However, the display correction scheme of existing AR glasses is to uniformly correct the displayed image indiscriminately, and is unable to perform different corrections for users in different situations, resulting in what the human eye sees is still unclear or distorted, affecting the user experience. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method and device, an electronic device and a storage medium, which can realize adaptive correction of the display image of the electronic device according to different eye movement data of the target object, so as to improve the display effect of the electronic device and the user experience.

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising at least:

[0007] Acquiring eye movement data of a target object gazing at an image displayed on an electronic device;

[0008] Based on the eye movement data, correcting the displayed image to obtain a target image;

[0009] Display is performed based on the target image.

[0010] In some embodiments, the correcting the displayed image based on the eye movement data to obtain a target image includes:

[0011] Determining a pupil distance between two pupils of the target object based on the eye movement data;

[0012] Based on the pupil distance, the displayed image is corrected to obtain the target image.

[0013] In some embodiments, the correcting the displayed image based on the pupil distance to obtain the target image includes:

[0014] Based on the pupil distance, performing distortion correction on the displayed image to obtain a distortion-corrected image;

[0015] Based on the pupil distance, performing dispersion correction on the distortion-corrected image to obtain a dispersion-corrected image;

[0016] Based on the pupil distance, shadow correction is performed on the dispersion-corrected image to obtain the target image.

[0017] In some embodiments, the performing distortion correction on the displayed image based on the pupil distance to obtain the distortion-corrected image includes:

[0018] Based on the pupil distance and the distortion type of the displayed image, determining a target distortion correction model that matches the pupil distance;

[0019] Based on the target distortion correction model, the display image is subjected to distortion correction to obtain the distortion corrected image.

[0020] In some embodiments, determining a target distortion correction model matching the pupil distance based on the pupil distance and the distortion type of the displayed image includes:

[0021] Determining distortion correction parameters based on a preset distortion parameter model and the pupil distance;

[0022] According to the distortion correction parameters, a plurality of initial distortion correction models are obtained;

[0023] Based on the distortion type of the displayed image, the target distortion correction model is determined from a plurality of the initial distortion correction models.

[0024] In some embodiments, determining the target distortion correction model from a plurality of the initial distortion correction models based on the distortion type of the displayed image comprises:

[0025] In a case where the distortion type of the displayed image is radial distortion, selecting the initial distortion correction model corresponding to the radial distortion from the multiple initial distortion correction models as the target distortion correction model;

[0026] In the case where the distortion type of the displayed image is tangential distortion, the initial distortion correction model corresponding to the tangential distortion is selected from the multiple initial distortion correction models as the target distortion correction model.

[0027] In some embodiments, the performing distortion correction on the displayed image based on the target distortion correction model to obtain the distortion corrected image includes:

[0028] Acquire a specified distance between a pixel point of a distorted area in the displayed image and a center point of the displayed image;

[0029] Inputting the designated distance and the coordinates of the pixel points in the distorted area into the target distortion correction model to obtain distortion correction coordinates;

[0030] Based on the distortion-corrected coordinates, the distortion-corrected image is determined.

[0031] In some embodiments, performing dispersion correction on the distortion-corrected image based on the pupil distance to obtain the dispersion-corrected image includes:

[0032] Based on the pupil distance, determining dispersion correction data matching the pupil distance;

[0033] Based on the dispersion correction data, dispersion correction is performed on the distortion-corrected image to obtain the dispersion-corrected image.

[0034] In some embodiments, the dispersion correction data includes dispersion correction color values ​​and dispersion correction coordinates.

[0035] In some embodiments, performing shadow correction on the dispersion-corrected image based on the pupil distance to obtain the target image includes:

[0036] Determining target brightness of pixels in a shadow area of ​​the dispersion-corrected image based on the pupil distance;

[0037] Based on the target brightness, shading correction is performed on the dispersion-corrected image to obtain the target image.

[0038] In some embodiments, determining the target brightness of the pixel points in the shadow area of ​​the dispersion-corrected image based on the pupil distance includes:

[0039] Obtaining the initial illumination intensity of the pixel points in the shadow area;

[0040] The target brightness is obtained based on the pupil distance, the initial light intensity and a preset mapping relationship; wherein the preset mapping relationship includes a mapping relationship between pupil distance, light intensity and brightness.

[0041] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, comprising at least:

[0042] An acquisition module configured to acquire eye movement data of a target object gazing at a display image of an electronic device;

[0043] A correction module, configured to correct the displayed image based on the eye movement data to obtain a target image;

[0044] The display module is configured to display based on the target image.

[0045] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising at least:

[0046] processor;

[0047] a memory for storing processor-executable instructions;

[0048] Wherein, the processor is configured to execute the image processing method as described in the first aspect.

[0049] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can perform the image processing method as described in the first aspect.

[0050] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0051] An image processing method provided by an embodiment of the present disclosure may include: obtaining eye movement data of a target object gazing at a display image of an electronic device; based on the eye movement data, correcting the display image to obtain a target image; and displaying based on the target image. Thus, unlike the related art which uniformly and indiscriminately corrects the display image, the embodiment of the present disclosure can perform different corrections on the display image of the electronic device according to different eye movement data of the target object, so as to obtain a target image that matches the eye movement data for display, thereby being able to realize adaptive correction of the display image of the electronic device according to different eye movement data of the target object, so as to improve the display effect of the electronic device and the user experience.

[0052] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0054] Figure 1 is a schematic diagram of a process of an image processing method according to an exemplary embodiment Figure 1 .

[0055] Figure 2It is a schematic diagram of pupil distance in an image processing method according to an exemplary embodiment.

[0056] Figure 3 is a schematic diagram of a process of an image processing method according to an exemplary embodiment Figure 2 .

[0057] Figure 4 is a schematic diagram of a process of an image processing method according to an exemplary embodiment Figure 3 .

[0058] Figure 5 is a schematic diagram of a process of an image processing method according to an exemplary embodiment Figure 4 .

[0059] Figure 6 The figure is a structural block diagram of an image processing device according to an exemplary embodiment.

[0060] Figure 7 The diagram is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0061] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0062] The technical solutions provided by various embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

[0063] In the related art, AR devices such as head-mounted devices such as AR glasses usually produce distortion on the display screen, which is specifically manifested as barrel distortion or pincushion distortion of the overall display image; dispersion occurs, which is manifested as stratification of the peripheral color system (Red Green Blue, RGB) of the display image; and shadow occurs, where the position of the human eye's gaze point is brighter but the periphery of the gaze point is darker.

[0064] However, the display correction scheme of existing AR glasses is to uniformly correct the displayed image indiscriminately, and is unable to perform different corrections for users in different situations, resulting in what the human eye sees is still unclear or distorted, affecting the user experience.

[0065] Based on this, an embodiment of the present disclosure provides an image processing method. Figure 1is a schematic diagram of a process of an image processing method according to an exemplary embodiment Figure 1 ,like Figure 1 As shown, the image processing method provided by the embodiment of the present disclosure includes at least the following steps:

[0066] Step 110: Acquire eye movement data of the target object staring at a display image of the electronic device;

[0067] Step 120: Correct the displayed image based on the eye movement data to obtain a target image;

[0068] Step 130: Display based on the target image.

[0069] In the embodiments of the present disclosure, the above-mentioned image processing method can be executed by the above-mentioned electronic device, wherein the above-mentioned electronic device may include: a mobile phone, a tablet computer, a smart watch, a digital camera, a head mounted display device (Head Mounted Display, HMD), etc., and the embodiments of the present disclosure are not limited.

[0070] For example, the above-mentioned electronic device may be an AR glasses. Each lens of the AR glasses is equipped with a high-definition projector and an implanted motion sensor. It can be regarded as a miniature mobile phone. It determines the user's current state by tracking the eye's line of sight to activate the corresponding function.

[0071] In step 110, the target object may be a user of the electronic device; for example, the target object may be a user wearing AR glasses.

[0072] The above-mentioned display image can be an image displayed on the display screen of the above-mentioned electronic device at the current moment; for example, the above-mentioned display image can be an image that produces distortion, dispersion or shadow phenomena in the AR glasses and needs to be corrected.

[0073] The above-mentioned eye movement data may be data related to the eyes of the above-mentioned target object; for example, the above-mentioned eye movement data may be the pupil distance between the two pupils of the user wearing the AR glasses, or the distance between the two gaze points of the two pupils of the user wearing the AR glasses on the above-mentioned displayed image, etc.

[0074] It can be understood that obtaining the eye movement data of the target object gazing at the display image of the electronic device may include: determining the eye movement data of the target object based on an eye tracking device in the electronic device.

[0075] Here, the eye tracking device may be a device that performs eye tracking using infrared technology, or may be a device that performs eye tracking using image acquisition, etc. For example, the eye tracking device may illuminate the eyes of the target object using infrared rays, and accurately capture the eye movement and pupil position of the target object by detecting the reflected infrared rays.

[0076] In the disclosed embodiment, when the electronic device is in working state, the electronic device can obtain the eye movement data of the target object in real time, so as to perform real-time correction on the displayed image subsequently, thereby improving the display effect of the electronic device and the user experience.

[0077] In step 120, the target image may be an image that needs to be displayed on the display screen of the electronic device at the next moment; for example, the target image may be an image obtained after correcting a display image that produces distortion, dispersion or shadow phenomena in the AR glasses.

[0078] In the embodiment of the present disclosure, correcting the displayed image based on the eye movement data to obtain the target image may include: determining the distance between two gaze points of two pupils of the target object on the displayed image based on the eye movement data, and then correcting the displayed image based on the distance between the two gaze points to obtain the target image; or determining the pupil distance between the two pupils of the target object based on the eye movement data, and then correcting the displayed image based on the pupil distance to obtain the target image.

[0079] Here, correcting the displayed image based on the eye movement data may include: performing distortion correction on the displayed image based on the eye movement data; and / or performing dispersion correction on the displayed image based on the eye movement data; and / or performing shadow correction on the displayed image based on the eye movement data.

[0080] In one embodiment of the present disclosure, step 120 may specifically include:

[0081] Determining a pupil distance between two pupils of the target object based on the eye movement data;

[0082] Based on the pupil distance, the displayed image is corrected to obtain the target image.

[0083] In this way, the pupil distance between the two pupils of the target object can be determined through eye movement data, so as to correct the displayed image to obtain the target image, thereby enabling adaptive correction of the displayed image of the electronic device for target objects with different pupil distances, thereby improving the display effect of the electronic device and the user experience.

[0084] The pupil distance may be the distance between two pupils of the target object; for example, the pupil distance may be 55 millimeters (mm), 58 mm, or 60 mm.

[0085] It can be understood that based on the above-mentioned eye movement data such as the position information of the eyes and pupils of the above-mentioned target object, the pupil distance between the two pupils of the above-mentioned target object is determined to perform distortion, dispersion and / or shadow correction on the above-mentioned display image to obtain the above-mentioned target image.

[0086] In one embodiment of the present disclosure, determining the pupil distance between the two pupils of the target object based on the eye movement data may specifically include: determining the current coordinates of the two pupils of the target object based on the eye movement data; and calculating the current coordinates of the two pupils to obtain the pupil distance.

[0087] Here, the current coordinates of the two pupils may be the position coordinates of the two pupils when the target object is gazing at the display image of the electronic device at the current moment.

[0088] It should be noted that an xy plane coordinate system may be established based on the center point of any one of the two eyes of the target object as the coordinate origin to determine the current coordinates of the two pupils.

[0089] For example, Figure 2 As shown, the electronic device can use the infrared technology in the eye tracking device to determine the eye movement and pupil position of the target object at the current moment, and then establish a coordinate system to obtain the current coordinates of the first pupil of the target object as (x1, y1), and the current coordinates of the second pupil as (x2, y2). The calculation formula of the pupil distance (PD) can be

[0090] It can be understood that by determining the pupil distance of the target object through the eye movement data, the displayed image can be firstly subjected to distortion correction based on the pupil distance to obtain a distortion-corrected image; then, dispersion correction can be performed on the distortion-corrected image based on the pupil distance to obtain a dispersion-corrected image; finally, shadow correction can be performed on the dispersion-corrected image based on the pupil distance to obtain the target image, thereby realizing adaptive correction of the displayed image of the electronic device.

[0091] In step 130, after the display image is corrected based on the eye movement data to obtain a target image, the electronic device may display the target image on a display screen, thereby achieving adaptive correction of the display image of the electronic device.

[0092] The image processing method provided by the embodiment of the present disclosure obtains the eye movement data of the target object gazing at the display image of the electronic device; based on the eye movement data, the display image is corrected to obtain the target image; and the display is performed based on the target image. Thus, unlike the related art which uniformly and indiscriminately corrects the display image, the embodiment of the present disclosure can perform different corrections on the display image of the electronic device according to the different eye movement data of the target object, so as to obtain the target image matching the eye movement data for display, thereby being able to realize adaptive correction of the display image of the electronic device according to the different eye movement data of the target object, so as to improve the display effect of the electronic device and the user experience.

[0093] In one embodiment of the present disclosure, in step 120, the displayed image is corrected based on the pupil distance to obtain the target image, which can be achieved in various different ways.

[0094] A specific implementation example is given below. It should be noted that the following is only an example and is not intended to be limiting.

[0095] Figure 3 is a schematic diagram of a process of an image processing method according to an exemplary embodiment Figure 2 See also Figure 3 In step 120 of the image processing method provided in the embodiment of the present disclosure, correcting the displayed image based on the pupil distance to obtain the target image may include:

[0096] Step 121: Based on the pupil distance, perform distortion correction on the displayed image to obtain a distortion-corrected image;

[0097] Step 122: Based on the pupil distance, perform dispersion correction on the distortion-corrected image to obtain a dispersion-corrected image;

[0098] Step 123: Based on the pupil distance, perform shadow correction on the dispersion-corrected image to obtain the target image.

[0099] In step 121, the distortion-corrected image may be a corrected image obtained after distortion correction is performed on the display image; that is, the distortion-corrected image has overcome the distortion phenomenon generated in the display image.

[0100] It can be understood that the embodiments of the present disclosure can determine a target distortion correction model that matches the pupil distance based on the pupil distance to perform distortion correction on the above-mentioned displayed image; or, it can also determine a mapping relationship between the pupil distance and the distortion correction coefficient based on the pupil distance to obtain the distortion correction coefficient that can perform distortion correction on the above-mentioned displayed image to obtain the above-mentioned distortion-corrected image; and so on.

[0101] In the disclosed embodiment, based on the pupil distance and the distortion type of the displayed image, a target distortion correction model matching the pupil distance can be selected from a plurality of initial distortion correction models, and then the displayed image can be distortion corrected based on the target distortion correction model to obtain the distortion-corrected image, thereby overcoming the distortion phenomenon generated in the displayed image.

[0102] In step 122, the dispersion-corrected image may be a corrected image obtained after dispersion correction is performed on the distortion-corrected image; that is, the dispersion-corrected image has overcome the distortion and dispersion phenomena generated in the display image.

[0103] Among them, the dispersion phenomenon can be a phenomenon in which a rainbow is formed at the focus of the lens due to the refractive index of the lens being related to the wavelength of light, that is, RGB colors are layered at the focus.

[0104] Here, the disclosed embodiment can determine the corresponding machine learning model for dispersion correction based on the above-mentioned pupil distance to obtain dispersion correction data matching the pupil distance, so as to perform dispersion correction on the distortion-corrected image; or, it can also determine the mapping relationship between the pupil distance and the dispersion correction color value and the mapping relationship between the pupil distance and the dispersion correction coordinates based on the above-mentioned pupil distance, so as to obtain the dispersion correction color value and dispersion correction coordinates that can perform dispersion correction on the above-mentioned distortion-corrected image, so as to obtain the above-mentioned dispersion-corrected image; and so on.

[0105] In one embodiment of the present disclosure, step 122 may specifically include:

[0106] Based on the pupil distance, determining dispersion correction data matching the pupil distance;

[0107] Based on the dispersion correction data, dispersion correction is performed on the distortion-corrected image to obtain the dispersion-corrected image.

[0108] In this way, the dispersion correction data matching the pupil distance can be determined by the pupil distance to perform dispersion correction on the distortion-corrected image to obtain a dispersion-corrected image, thereby overcoming the distortion and dispersion phenomena generated in the displayed image and preparing for the subsequent corrected target image.

[0109] The dispersion correction data may include a dispersion correction color value and a dispersion correction coordinate. The dispersion correction color value may be a color value displayed after dispersion correction is performed on an initial color value of a pixel point in a dispersion area in the distortion-corrected image; and the dispersion correction coordinate may be a position coordinate displayed after dispersion correction is performed on an initial coordinate of a pixel point in a dispersion area in the distortion-corrected image.

[0110] It can be understood that starting from the two perspectives of the color value and coordinates of the pixel points in the dispersion area in the above-mentioned distortion-corrected image, the dispersion-corrected color value and the dispersion-corrected coordinates matching the pupil distance are determined based on the above-mentioned pupil distance, so as to perform dispersion correction on the color value and coordinates of the pixel points in the dispersion area in the above-mentioned distortion-corrected image, thereby overcoming the dispersion phenomenon in the distortion-corrected image.

[0111] In step 123, the shadow-corrected image may be a corrected image obtained by performing shadow correction on the dispersion-corrected image; that is, the shadow-corrected image has overcome the distortion, dispersion and shadow phenomena generated in the display image.

[0112] In the disclosed embodiments, a machine learning model corresponding to shadow correction can be determined based on the pupil distance to obtain a target brightness that matches the pupil distance, so as to perform shadow correction on the dispersion-corrected image; or, a mapping relationship between pupil distance and brightness can be determined based on the pupil distance to obtain a target brightness that can perform shadow correction on the dispersion-corrected image, so as to obtain the target image; and so on.

[0113] In some embodiments, step 123 may specifically include:

[0114] Determining target brightness of pixels in a shadow area of ​​the dispersion-corrected image based on the pupil distance;

[0115] Based on the target brightness, shading correction is performed on the dispersion-corrected image to obtain the target image.

[0116] In this way, the target brightness of the pixel points in the shadow area of ​​the dispersion-corrected image can be determined by the pupil distance, so as to perform shadow correction on the dispersion-corrected image to obtain the target image, thereby overcoming the distortion, dispersion and shadow phenomena generated in the displayed image and realizing adaptive correction of the displayed image of the electronic device.

[0117] The shadow area may be an area in the dispersion-corrected image where a shadow phenomenon occurs; and the target brightness may be a brightness value displayed after shadow correction is performed on the initial brightness of the pixel points in the shadow area in the dispersion-corrected image.

[0118] It can be understood that the above-mentioned electronic device can directly obtain the initial brightness of the pixel points in the above-mentioned shadow area, and then obtain the above-mentioned target brightness based on the above-mentioned pupil distance, the above-mentioned initial brightness and the mapping relationship between the pupil distance and the brightness, so as to perform shadow correction on the above-mentioned dispersion-corrected image to obtain the above-mentioned target image; or, the above-mentioned electronic device can also obtain the initial light intensity of the pixel points in the above-mentioned shadow area, and then obtain the above-mentioned target brightness based on the above-mentioned pupil distance, the above-mentioned initial light intensity and the mapping relationship between the pupil distance, the light intensity and the brightness, so as to perform shadow correction on the above-mentioned dispersion-corrected image to obtain the above-mentioned target image; and so on.

[0119] In one embodiment of the present disclosure, determining the target brightness of the pixel points in the shadow area of ​​the dispersion-corrected image based on the pupil distance includes:

[0120] Obtaining the initial illumination intensity of the pixel points in the shadow area;

[0121] The target brightness is obtained based on the pupil distance, the initial light intensity and a preset mapping relationship; wherein the preset mapping relationship includes a mapping relationship between pupil distance, light intensity and brightness.

[0122] In this way, the target brightness can be determined by the pupil distance, the initial light intensity of the pixel points in the shadow area of ​​the dispersion-corrected image, and the preset mapping relationship between the pupil distance, light intensity and brightness, so as to prepare for the subsequent shadow correction of the dispersion-corrected image, thereby achieving shadow correction of the dispersion-corrected image and improving the display effect of the electronic device.

[0123] The initial illumination intensity may be the illumination intensity of the pixels in the shadow area before shadow correction is performed. It should be noted that the initial illumination intensity may be measured based on the ambient light sensor in the electronic device.

[0124] The above-mentioned preset mapping relationship may include a mapping relationship between pupil distance and light intensity, a mapping relationship between pupil distance and brightness, and a mapping relationship between light intensity and brightness.

[0125] In the disclosed embodiment, based on the pupil distance, a mapping relationship between light intensity and brightness that matches the pupil distance can be determined from a plurality of mapping relationship lookup tables between light intensity and brightness; then, based on the initial light intensity and the mapping relationship between the light intensity and brightness, the target brightness of the pixel points in the shadow area can be determined, so as to perform shadow correction on the dispersion-corrected image to obtain the target image, thereby overcoming the distortion, dispersion and shadow phenomena generated in the displayed image.

[0126] The image processing method provided by the embodiments of the present disclosure can perform distortion, dispersion and shadow correction on the display image of the electronic device in sequence based on the pupil distance of the target object, so as to obtain a target image that matches the pupil distance for display, thereby being able to achieve adaptive correction of the display image of the electronic device for different pupil distances of the target object, so as to improve the display effect of the electronic device and the user experience.

[0127] In one embodiment of the present disclosure, step 121 may be implemented in various ways.

[0128] A specific implementation example is given below. It should be noted that the following is only an example and is not intended to be limiting.

[0129] Figure 4 is a schematic diagram of a process of an image processing method according to an exemplary embodiment Figure 3 See also Figure 4 , step 121 in the image processing method provided by the embodiment of the present disclosure may include:

[0130] Step 121a: determining a target distortion correction model matching the pupil distance based on the pupil distance and the distortion type of the displayed image;

[0131] Step 121b: Based on the target distortion correction model, perform distortion correction on the displayed image to obtain the distortion corrected image.

[0132] In step 121a, the distortion types of the displayed image may include radial distortion and tangential distortion, wherein radial distortion may represent distortion caused by image enlargement or reduction, such as barrel or pincushion distortion, etc., and tangential distortion may represent distortion caused by image displacement.

[0133] The above-mentioned target distortion correction model may include a machine learning model about the coordinates of pixel points before and after distortion correction, or a functional relationship about the coordinates of pixel points before and after distortion correction, etc.

[0134] In some embodiments, step 121a may specifically include:

[0135] Determining distortion correction parameters based on a preset distortion parameter model and the pupil distance;

[0136] According to the distortion correction parameters, a plurality of initial distortion correction models are obtained;

[0137] Based on the distortion type of the displayed image, the target distortion correction model is determined from a plurality of the initial distortion correction models.

[0138] In this way, the distortion correction parameters can be first determined through the pupil distance and the distortion parameter model, and then multiple initial distortion correction models can be obtained according to the distortion correction parameters. Then, based on the distortion type of the displayed image, the target distortion correction model can be determined from the multiple initial distortion correction models, thereby preparing for the subsequent distortion correction of the displayed image, realizing the distortion correction of the displayed image, and improving the display effect of the electronic device.

[0139] The distortion parameter model may be a model pre-trained based on multiple sets of sample pupil distances and sample distortion correction parameters. The initial distortion correction model may be all distortion correction models corresponding to various distortion types of the displayed image. The distortion correction parameters may be correction parameters in the initial distortion correction model.

[0140] It should be noted that the above-mentioned distortion parameter model can obtain multiple groups of sample data from a training data set in which the distortion correction parameters have been determined, each group of sample data includes a sample pupil distance and a sample distortion correction parameter corresponding to the sample pupil distance; the initial distortion parameter model is pre-trained by using the multivariate linear regression method to obtain the above-mentioned distortion parameter model.

[0141] Exemplarily, assuming that the pupil distance is x and the distortion correction parameter is y, the above distortion parameter model can be expressed as: y=ax+b; wherein a and b respectively represent the slope and intercept of the distortion parameter model, and a and b can be obtained by performing least squares solution on multiple groups of sample pupil distances and sample distortion correction parameters, so that the pupil distance can be subsequently input into the above distortion parameter model to obtain the distortion correction parameter corresponding to the pupil distance.

[0142] In one embodiment of the present disclosure, determining the target distortion correction model from a plurality of initial distortion correction models based on the distortion type of the displayed image includes:

[0143] In a case where the distortion type of the displayed image is radial distortion, selecting the initial distortion correction model corresponding to the radial distortion from the multiple initial distortion correction models as the target distortion correction model;

[0144] In the case where the distortion type of the displayed image is tangential distortion, the initial distortion correction model corresponding to the tangential distortion is selected from the multiple initial distortion correction models as the target distortion correction model.

[0145] In this way, by using different distortion types of the displayed image, a target distortion correction model corresponding to the distortion type can be selected from multiple initial distortion correction models, thereby preparing for subsequent distortion correction of the displayed image, realizing distortion correction of the displayed image, and improving the display effect of the electronic device.

[0146] It can be understood that the target distortion correction model determined when the distortion type of the above-mentioned displayed image is radial distortion may be different from the target distortion correction model determined when the distortion type of the above-mentioned displayed image is tangential distortion; that is, the distortion correction parameters in the two target distortion correction models may also be different.

[0147] In step 121b, the coordinates of the pixel points in the distorted area in the above-mentioned display image are input into the above-mentioned target distortion correction model to obtain the distortion correction coordinates of the pixel points in the distorted area, and then the above-mentioned display image is distortion corrected based on the distortion correction coordinates to obtain the above-mentioned distortion corrected image.

[0148] In one embodiment of the present disclosure, step 121b may specifically include:

[0149] Acquire a specified distance between a pixel point of a distorted area in the displayed image and a center point of the displayed image;

[0150] Inputting the designated distance and the coordinates of the pixel points in the distorted area into the target distortion correction model to obtain distortion correction coordinates;

[0151] Based on the distortion-corrected coordinates, the distortion-corrected image is determined.

[0152] In this way, the distortion correction coordinates can be obtained by inputting the specified distance between the pixel point of the distorted area in the display image and the center point of the display image and the coordinates of the pixel point in the distorted area into the target distortion correction model to obtain the distortion corrected image, thereby overcoming the distortion phenomenon generated in the display image and preparing for the subsequent dispersion correction to obtain the dispersion corrected image.

[0153] Among them, the above-mentioned distortion area can be the area where the distortion phenomenon occurs in the above-mentioned display image; the above-mentioned specified distance can be the distance between the pixel points of the above-mentioned distortion area and the center point of the above-mentioned display image; the above-mentioned distortion correction coordinates can be the position coordinates of the pixel points of the above-mentioned distortion area after distortion correction.

[0154] In the embodiment of the present disclosure, the coordinates of the pixel point in the above-mentioned distortion area may include the horizontal coordinate and the vertical coordinate of the pixel point before distortion correction; correspondingly, the above-mentioned distortion correction coordinates may include the horizontal coordinate and the vertical coordinate of the pixel point after distortion correction.

[0155] For example, when the distortion type of the above-mentioned displayed image is radial distortion, the determined target distortion correction model may be formula (1).

[0156]

[0157] In the case where the distortion type of the above-mentioned displayed image is tangential distortion, the determined target distortion correction model may be formula (2).

[0158]

[0159] Among them, (x 0 ,y 0 ) can be the coordinates of the pixel points in the above-mentioned distortion area; (x, y) can be the above-mentioned distortion correction coordinates; r can be the above-mentioned specified distance; k1, k2, k3, p1, p2 can be the above-mentioned distortion correction parameters.

[0160] The image processing method provided by the embodiment of the present disclosure can first determine the distortion correction parameters through the pupil distance and the distortion parameter model, then obtain multiple initial distortion correction models according to the distortion correction parameters, and then determine the target distortion correction model from the multiple initial distortion correction models based on the distortion type of the displayed image. Finally, the distortion correction coordinates are obtained by inputting the specified distance between the pixel points of the distorted area in the displayed image and the center point of the displayed image and the coordinates of the pixel points in the distorted area into the target distortion correction model to obtain the distortion corrected image, so as to achieve the distortion correction of the displayed image of the electronic device for different pupil distances of the target object, so as to improve the display effect of the electronic device and the user experience.

[0161] Figure 5 is a schematic diagram of a process of an image processing method according to an exemplary embodiment Figure 4 .like Figure 5 As shown, the image processing method provided in the embodiment of the present disclosure is only an example and not a limitation, and is intended to facilitate those skilled in the art to better understand the technical solution of the present disclosure. Figure 5 The image processing method provided by the embodiment of the present disclosure may specifically include the following steps:

[0162] Step 501: Acquire eye movement data of a target object gazing at a display image of an electronic device;

[0163] Step 502: Determine the pupil distance between two pupils of the target object based on the eye movement data;

[0164] Step 503: determining distortion correction parameters based on a preset distortion parameter model and the pupil distance;

[0165] Step 504: obtaining a plurality of initial distortion correction models according to the distortion correction parameters;

[0166] Step 505: determining a target distortion correction model from a plurality of initial distortion correction models based on the distortion type of the displayed image;

[0167] Step 506: Obtain a specified distance between a pixel point in a distorted area in the displayed image and a center point of the displayed image;

[0168] Step 507: input the designated distance and the coordinates of the pixel points in the distorted area into the target distortion correction model to obtain distortion correction coordinates;

[0169] Step 508: Determine a distortion-corrected image based on the distortion-corrected coordinates;

[0170] Step 509: Based on the pupil distance, determine dispersion correction data matching the pupil distance;

[0171] Step 510: Perform dispersion correction on the distortion-corrected image based on the dispersion correction data to obtain a dispersion-corrected image;

[0172] Step 511, obtaining the initial illumination intensity of the pixel points in the shadow area of ​​the dispersion-corrected image;

[0173] Step 512: obtaining a target brightness based on the pupil distance, the initial illumination intensity and a preset mapping relationship;

[0174] Step 513: Based on the target brightness, perform shadow correction on the dispersion-corrected image to obtain a target image;

[0175] Step 514: Display based on the target image.

[0176] The image processing method provided by the embodiment of the present disclosure can first determine the distortion correction parameters through the pupil distance and the distortion parameter model, then obtain multiple initial distortion correction models according to the distortion correction parameters, and then determine the target distortion correction model from the multiple initial distortion correction models based on the distortion type of the displayed image, and finally obtain the distortion correction coordinates by inputting the specified distance between the pixel points of the distorted area in the displayed image and the center point of the displayed image and the coordinates of the pixel points in the distorted area into the target distortion correction model to obtain the distortion corrected image; then, the distortion corrected image is sequentially subjected to dispersion and shadow correction to obtain the target image matching the pupil distance for display, so that the display image of the electronic device can be adaptively corrected for different pupil distances of the target object, so as to improve the display effect of the electronic device and the user experience.

[0177] Figure 6 This is a structural block diagram of an image processing device provided by an embodiment of the present disclosure. Figure 6 The image processing device 600 provided in the embodiment of the present disclosure may include: an acquisition module 610, a correction module 620 and a display module 630.

[0178] The acquisition module 610 is configured to acquire eye movement data of a target object gazing at a display image of an electronic device;

[0179] A correction module 620 is configured to correct the displayed image based on the eye movement data to obtain a target image;

[0180] The display module 630 is configured to display based on the target image.

[0181] The image processing device provided in the embodiment of the present disclosure obtains the eye movement data of the target object gazing at the display image of the electronic device; based on the eye movement data, the display image is corrected to obtain the target image; and the display is performed based on the target image. Thus, unlike the related art which uniformly and indiscriminately corrects the display image, the embodiment of the present disclosure can perform different corrections on the display image of the electronic device according to the different eye movement data of the target object, so as to obtain the target image matching the eye movement data for display, thereby being able to realize adaptive correction of the display image of the electronic device according to the different eye movement data of the target object, so as to improve the display effect of the electronic device and the user experience.

[0182] for Figure 6 In a possible implementation of the technical solution shown, the correction module 620 can be specifically configured to: determine the pupil distance between the two pupils of the target object based on the eye movement data; and correct the displayed image based on the pupil distance to obtain the target image.

[0183] for Figure 6 In the technical solution shown, in a possible implementation, the correction module 620 can also be configured as: based on the pupil distance, performing distortion correction on the displayed image to obtain a distortion-corrected image; based on the pupil distance, performing dispersion correction on the distortion-corrected image to obtain a dispersion-corrected image; based on the pupil distance, performing shadow correction on the dispersion-corrected image to obtain the target image.

[0184] for Figure 6 In the technical solution shown, in a possible implementation, the correction module 620 can also be configured to: determine a target distortion correction model that matches the pupil distance based on the pupil distance and the distortion type of the displayed image; and perform distortion correction on the displayed image based on the target distortion correction model to obtain the distortion corrected image.

[0185] for Figure 6In the technical solution shown, in a possible implementation method, the correction module 620 can also be configured as: determining the distortion correction parameters based on a preset distortion parameter model and the pupil distance; obtaining multiple initial distortion correction models according to the distortion correction parameters; and determining the target distortion correction model from the multiple initial distortion correction models based on the distortion type of the displayed image.

[0186] for Figure 6 In the technical solution shown, in a possible implementation method, the correction module 620 can also be configured as: when the distortion type of the displayed image is radial distortion, the initial distortion correction model corresponding to the radial distortion is selected from the multiple initial distortion correction models as the target distortion correction model; when the distortion type of the displayed image is tangential distortion, the initial distortion correction model corresponding to the tangential distortion is selected from the multiple initial distortion correction models as the target distortion correction model.

[0187] for Figure 6 In the technical solution shown, in a possible implementation method, the correction module 620 can also be configured to: obtain a specified distance between a pixel point in the distorted area in the displayed image and a center point of the displayed image; input the specified distance and the coordinates of the pixel point in the distorted area into the target distortion correction model to obtain distortion correction coordinates; and determine the distortion corrected image based on the distortion correction coordinates.

[0188] for Figure 6 In the technical solution shown, in a possible implementation, the correction module 620 can also be configured to: determine, based on the pupil distance, dispersion correction data that matches the pupil distance; and perform dispersion correction on the distortion-corrected image based on the dispersion correction data to obtain the dispersion-corrected image.

[0189] for Figure 6 In the technical solution shown, in a possible implementation manner, the dispersion correction data includes dispersion correction color values ​​and dispersion correction coordinates.

[0190] for Figure 6 In the technical solution shown, in a possible implementation, the correction module 620 can also be configured to: determine the target brightness of the pixel points in the shadow area of ​​the dispersion-corrected image based on the pupil distance; and perform shadow correction on the dispersion-corrected image based on the target brightness to obtain the target image.

[0191] for Figure 6In the technical solution shown, in a possible implementation, the correction module 620 can also be configured to: obtain the initial light intensity of the pixel points in the shadow area; obtain the target brightness based on the pupil distance, the initial light intensity and a preset mapping relationship; wherein the preset mapping relationship includes a mapping relationship between pupil distance, light intensity and brightness.

[0192] It should be noted that Figure 6 The specific manner in which each module of the image processing device in the illustrated embodiment performs operations has been described in detail in the embodiment of the method, and will not be elaborated on here.

[0193] Figure 7 8 is a schematic diagram of a structure of an electronic device according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, a wearable device, etc.

[0194] Reference Figure 7 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0195] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0196] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0197] The power component 806 provides power to the various components of the electronic device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0198] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0199] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0200] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0201] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0202] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as Wi-Fi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0203] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0204] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0205] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform an image processing method, the method comprising: obtaining eye movement data of a target object gazing at a display image of the electronic device; based on the eye movement data, correcting the display image to obtain a target image; and displaying based on the target image.

[0206] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0207] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0208] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquiring eye movement data of a target object gazing at an image displayed on an electronic device; Based on the eye movement data, correcting the displayed image to obtain a target image; Display is performed based on the target image.

2. The method according to claim 1, characterized in that The step of correcting the displayed image based on the eye movement data to obtain a target image includes: Determining a pupil distance between two pupils of the target object based on the eye movement data; Based on the pupil distance, the displayed image is corrected to obtain the target image.

3. The method according to claim 2, characterized in that The method of correcting the displayed image based on the pupil distance to obtain the target image includes: Based on the pupil distance, performing distortion correction on the displayed image to obtain a distortion-corrected image; Based on the pupil distance, performing dispersion correction on the distortion-corrected image to obtain a dispersion-corrected image; Based on the pupil distance, shadow correction is performed on the dispersion-corrected image to obtain the target image.

4. The method according to claim 3, characterized in that The step of performing distortion correction on the displayed image based on the pupil distance to obtain a distortion-corrected image includes: Based on the pupil distance and the distortion type of the displayed image, determining a target distortion correction model that matches the pupil distance; Based on the target distortion correction model, the display image is subjected to distortion correction to obtain the distortion corrected image.

5. The method according to claim 4, characterized in that The step of determining a target distortion correction model matching the pupil distance based on the pupil distance and the distortion type of the displayed image includes: Determining distortion correction parameters based on a preset distortion parameter model and the pupil distance; According to the distortion correction parameters, a plurality of initial distortion correction models are obtained; Based on the distortion type of the displayed image, the target distortion correction model is determined from a plurality of the initial distortion correction models.

6. The method according to claim 5, characterized in that The determining the target distortion correction model from the plurality of initial distortion correction models based on the distortion type of the displayed image comprises: In a case where the distortion type of the displayed image is radial distortion, selecting the initial distortion correction model corresponding to the radial distortion from the multiple initial distortion correction models as the target distortion correction model; In the case where the distortion type of the displayed image is tangential distortion, the initial distortion correction model corresponding to the tangential distortion is selected from the multiple initial distortion correction models as the target distortion correction model.

7. The method according to claim 4, characterized in that The step of performing distortion correction on the displayed image based on the target distortion correction model to obtain the distortion corrected image includes: Acquire a specified distance between a pixel point of a distorted area in the displayed image and a center point of the displayed image; Inputting the designated distance and the coordinates of the pixel points in the distorted area into the target distortion correction model to obtain distortion correction coordinates; Based on the distortion-corrected coordinates, the distortion-corrected image is determined.

8. The method according to claim 3, characterized in that The step of performing dispersion correction on the distortion-corrected image based on the pupil distance to obtain a dispersion-corrected image includes: Based on the pupil distance, determining dispersion correction data matching the pupil distance; Based on the dispersion correction data, dispersion correction is performed on the distortion-corrected image to obtain the dispersion-corrected image.

9. The method according to claim 8, characterized in that The dispersion correction data includes dispersion correction color values ​​and dispersion correction coordinates.

10. The method according to claim 3, characterized in that: The step of performing shadow correction on the dispersion-corrected image based on the pupil distance to obtain the target image comprises: Determining target brightness of pixels in a shadow area of ​​the dispersion-corrected image based on the pupil distance; Based on the target brightness, shading correction is performed on the dispersion-corrected image to obtain the target image.

11. The method according to claim 10, characterized in that The step of determining target brightness of pixel points in a shadow area of ​​the dispersion-corrected image based on the pupil distance includes: Obtaining the initial illumination intensity of the pixel points in the shadow area; The target brightness is obtained based on the pupil distance, the initial light intensity and a preset mapping relationship; wherein the preset mapping relationship includes a mapping relationship between pupil distance, light intensity and brightness.

12. An image processing device, characterized in that: include: An acquisition module configured to acquire eye movement data of a target object gazing at a display image of an electronic device; A correction module, configured to correct the displayed image based on the eye movement data to obtain a target image; The display module is configured to display based on the target image.

13. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the image processing method according to any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image processing method as claimed in any one of claims 1 to 11.

Images