Image processing method and device based on eyeball tracking, equipment and storage medium

Abstract

The embodiment of the invention discloses an image processing method and device based on eyeball tracking, equipment and a storage medium. Comprising the following steps: synchronously obtaining a sight direction and / or fixation point information of a user and an image watched by the user; dividing the image into a plurality of areas based on the sight line direction and / or fixation point information, wherein the number of the areas is at least two; and coding the image based on the plurality of regions and coding information, wherein the coding information comprises at least one of the following items: an overall target code rate and a quantization step size. According to the image processing method based on eyeball tracking provided by the embodiment of the invention, the environment image is coded based on the divided regions and the coding information, so that the image transmission quantity is reduced, the transmission becomes more convenient, and the data transmission speed is accelerated.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of eye tracking technology, and in particular to an image processing method, device, equipment and storage medium based on eye tracking. Background Art

[0002] Existing wearable devices with eye tracking functions include analytical glasses, virtual reality devices (VR), and augmented reality devices (AR). Analytical glasses and AR usually project the gaze point of the human eye onto the picture taken by the front camera through the combination of an eye tracking camera and a front camera, which is convenient for subsequent eye tracking experiments and analysis and human-computer interaction, such as human psychology and cognitive science research, medical auxiliary diagnosis, user experience analysis, user interaction, etc. Due to bandwidth limitations, the image frame rate collected by the front camera is lower than the image frame rate collected by the eye tracking camera. In this way, during the gaze point projection process, the frame rate of the gaze point needs to be reduced, and the data utilization rate is incomplete, which makes it difficult to use in eye tracking analysis experiments with high frame rate requirements.

[0003] Virtual reality (VR) devices generally have eye tracking functions. Users wear VR to view images on the screen and experience interactive activities such as games, watching movies and TV shows, watching live broadcasts, and holding video conferences. VR devices can capture and calculate the user's gaze point in real time and display it on the screen. Summary of the invention

[0004] The embodiment of the present invention provides an image processing method, device, equipment and storage medium based on eye tracking, which encodes the image that the user is looking at according to the divided multiple areas and coding information to reduce the size of the image transmission amount, thereby making the transmission more convenient and speeding up the data transmission speed. It also brings the possibility of subsequently improving the acquisition frame rate of the image acquisition device or the display frame rate of the image display device, thereby improving the accuracy of eye tracking.

[0005] In a first aspect, an embodiment of the present invention provides an image processing method based on eye tracking, comprising:

[0006] Synchronously obtaining the user's line of sight direction and / or gaze point information and the image the user is gazing at;

[0007] Dividing the image into a plurality of regions based on the sight line direction and / or gaze point information, wherein the plurality of regions is at least two;

[0008] The image is encoded based on the multiple regions and encoding information, where the encoding information includes at least one of the following: an overall target bit rate and a quantization step size.

[0009] In a second aspect, an embodiment of the present invention further provides an image processing device based on eye tracking, comprising:

[0010] An acquisition module, used for synchronously acquiring the user's sight direction and / or gaze point information and the image the user is gazing at;

[0011] A region division module, used for dividing the image into a plurality of regions based on the sight line direction and / or gaze point information, wherein the plurality of regions is at least two;

[0012] The image encoding module is used to encode the image based on the multiple regions and encoding information, wherein the encoding information includes at least one of the following: an overall target bit rate and a quantization step size.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the image processing method based on eye tracking described in an embodiment of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image processing method based on eye tracking described in the embodiment of the present invention when executed.

[0018] The embodiment of the present invention discloses an image processing method, device, equipment and storage medium based on eye tracking. The user's line of sight direction and / or gaze point information and the image that the user is gazing at are synchronously acquired; the image is divided into multiple regions based on the line of sight direction and / or gaze point information, and the multiple regions are at least two; the image is encoded based on the multiple regions and coding information, and the coding information includes at least one of the following: overall target bit rate, quantization step size. The image processing method based on eye tracking provided by the embodiment of the present invention encodes the image based on the divided multiple regions and coding information to reduce the size of the image transmission amount, thereby making the transmission more convenient and speeding up the data transmission speed. It also brings the possibility of subsequently increasing the acquisition frame rate of the image acquisition device or increasing the display frame rate of the image display device, and finally improving the accuracy of eye tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of an image processing method based on eye tracking in Embodiment 1 of the present invention;

[0020] Figure 2is an example diagram of dividing an image into multiple regions in the first embodiment of the present invention;

[0021] Figure 3 is an example diagram of dividing an image into multiple regions in the first embodiment of the present invention;

[0022] Figure 4 is an example diagram of projecting a gaze point onto an image in the first embodiment of the present invention;

[0023] Figure 5 is a structural schematic diagram of an image processing device based on eye tracking in Embodiment 2 of the present invention;

[0024] Figure 6 It is a structural schematic diagram of an electronic device in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0026] Embodiment 1

[0027] Figure 1 This is a flowchart of an image processing method based on eye tracking provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of encoding an environmental image in eye tracking. The method can be executed by an image processing device based on eye tracking, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server, etc. Specifically, it includes the following steps:

[0028] S110, synchronously obtaining the user's line of sight direction and / or gaze point information and the image the user is gazing at.

[0029] The line of sight direction and / or gaze point information can be determined based on the collected eye tracking data, and the eye tracking data can be collected by a first collection device. The image that the user is looking at can be an environmental image or an image displayed on a display screen. The environmental image can be collected by a second collection device.

[0030] Among them, the first acquisition device may include at least one of the following: an image sensor, a myoelectric biopotential sensor and an infrared scanner, and the eye tracking data may include at least one of the following: an eye image, myoelectric biopotential data and infrared scanning data.

[0031] Optionally, a method of obtaining the user's line of sight direction and / or gaze point information may be: obtaining eye tracking data, and determining the user's line of sight direction and / or gaze point information according to the eye tracking data.

[0032] In this application scenario, the line of sight and / or fixation point of the eye can be calculated and estimated based on the eye tracking data. According to existing knowledge, there are many ways to track the eye, such as pupil-corneal reflection method, pupil tracking method, biopotential method, MEMS infrared scanning method, etc. Different eye tracking methods correspond to different first acquisition devices; different first acquisition devices also correspond to different eye tracking data. Taking the pupil-corneal reflection method as an example, the first acquisition device can be an image sensor, and the eye tracking data can include eye images. When calculating the line of sight and / or fixation point of the eye, feature points are first extracted from the eye image to obtain eye features, wherein the eye features can include: pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, light spot (also called Purkin spot) position, etc. As the eyeball rotates, the relative position relationship between the pupil center and the light spot changes accordingly, and the correspondingly collected several eye images with light spots reflect such a position change relationship; the line of sight / fixation point estimation is performed based on the position change relationship.

[0033] Among them, the second acquisition device can be an image sensor. In this embodiment, since the first acquisition device is used to collect the user's eye tracking data, the amount of data occupied by the eye tracking data is small and will not be limited by bandwidth, so the acquisition frame rate of the first acquisition device is usually high, for example, it can be 120Hz or 240Hz. The image that the user is looking at, either the environmental image or the image displayed on the display screen, occupies a large amount of data and will be limited by bandwidth. Therefore, the acquisition frame rate of the second acquisition device or the display frame rate of the image display device is usually low, for example, it can be 30Hz or 60Hz. In this way, in the process of projecting the gaze point to the image, it is necessary to reduce the frame rate of the gaze point (a part of the gaze point will be deleted), and the utilization rate of the gaze point data is low, which is difficult to use in eye movement analysis experiments with high frame rate requirements, as well as AR and VR devices. Therefore, this embodiment needs to process the image that the user is looking at, so that the transmission becomes more convenient and the data transmission speed is accelerated.

[0034] S120, dividing the image gazed by the user into multiple areas based on the gaze direction and / or gaze point information;

[0035] The number of regions is at least two. The process of dividing the image into multiple regions based on the sight direction and / or the gaze point information may be: mapping the sight direction and / or the gaze point information into the image; dividing the image into multiple regions with the gaze point as the center and the edge of the graphic as the boundary.

[0036] The set graphic may be a circle, an ellipse, a rectangle, a square, a triangle, a polygon, a star, or an irregular shape, which is not limited here. Specifically, dividing the image into multiple areas with the gaze point as the center and the edge of the set graphic as the boundary can be understood as: in the image that the user is gazing at, multiple set graphics with different areas centered on the gaze point are drawn, and the areas surrounded by the edges of adjacent set graphics are the divided areas.

[0037] For example, assuming that the set figure is a circle and is divided into N areas, the first N-1 areas (sorted from small to large in terms of distance from the gaze point) are concentric circles or concentric rings with the gaze point as a circle, and the Nth area is the remaining area. Figure 2 is an example diagram of dividing an image into multiple regions in this embodiment. Figure 2 As shown, the graphic is set to be a circle, and the image is divided into three areas. The first area is a circular area with a radius of D1 pixels and a gaze point as the center. The second area is a circular area between the radius D1 and D2. The third area is an area with a radius greater than D2.

[0038] Exemplarily, assuming that the graphic is set to be a square and divided into N areas, the first N-1 areas (sorted in ascending order of distance from the gaze point) are square areas or U-shaped areas centered on the gaze point, and the Nth area is the remaining area. Figure 3 is an example diagram of dividing an image into multiple regions in this embodiment. Figure 3 As shown, the graphic is set to a square, and the image is divided into three areas. The first area is a square area with a side length of 2*D1 centered at the gaze point, the second area is a U-shaped area between squares with side lengths of 2*D1 and 2*D2, and the third area is the remaining area.

[0039] S130, encoding the image based on the multiple regions and the encoding information.

[0040] The coding information includes at least one of the following: the overall target bit rate, the quantization step size, and the sub-target coding rates of multiple regions. The bit rate can be understood as the amount of data transmitted per unit time, and the quantization step size is a quantization parameter, which can be understood as the amount of data between two adjacent encodings, and the amount of data can be represented by bits. In this application scenario, the coding information can be set by the user. For example: the overall target bit rate is set to 5000 bits / second.

[0041] In this embodiment, the manner of encoding the image based on the multiple regions and the encoding information may be: determining sub-target coding rates of the multiple regions according to the overall target coding rate; and encoding the multiple regions according to the sub-target coding rates of the multiple regions.

[0042] If the encoding information directly includes sub-target bit rates of multiple regions, the multiple regions are encoded respectively according to the sub-target bit rates of the multiple regions.

[0043] Optionally, the method of respectively determining the sub-target bit rates of the multiple regions according to the coding information may be: obtaining allocation ratios corresponding to the multiple regions, and determining the sub-target bit rates included in the multiple regions according to the overall target bit rate and the allocation ratio.

[0044] The allocation ratio is inversely proportional to the distance between the multiple regions and the gaze point. That is, the closer the distance to the gaze point, the greater the target bit rate allocated. The allocation ratio can be set by the user according to the distance between each region and the gaze point. For example, Figure 2 and Figure 3 For example, the distances between the first area, the second area and the third area and the gaze point gradually increase, and the allocation ratio can be set to: the allocation ratio of the first area, the second area and the third area can be: 5:2:1, that is, the sub-target bit rate of the first area is the largest, the sub-target bit rate of the second area is the second, and the sub-target bit rate of the third area is the smallest, so as to ensure the clarity of the area closer to the gaze point in the image that the user is looking at.

[0045] Specifically, after obtaining the sub-target code rates of multiple regions, the multiple regions are encoded respectively according to the sub-target code rates, so that the data transmission volume per unit time of each region reaches the corresponding target sub-code rate. Exemplarily, assuming that the overall target code rate in the encoding information is m, the allocation ratio of the first region, the second region and the third region is 5:2:1, the sub-target code rate allocated to the first region is m*5 / 8, the sub-target code rate allocated to the second region is m*3 / 8; and the sub-target code rate allocated to the third region is m*1 / 8. Finally, the first region is encoded so that the code rate of the encoded first region is m*5 / 8; the second region is encoded so that the code rate of the encoded second region is m*3 / 8; and the third region is encoded so that the code rate of the encoded third region is m*1 / 8.

[0046] Optionally, when the encoding information is a quantization step, a method of encoding the image based on multiple regions and the encoding information may be: obtaining quantization steps corresponding to the multiple regions; and encoding the multiple regions respectively according to the quantization steps of the multiple regions.

[0047] The quantization step length is proportional to the distance between the multiple regions and the gaze point, that is, the closer the distance to the gaze point, the smaller the quantization step length, and the smaller the amount of data per quantization encoding. The quantization step length of each region can be set by the user. In this embodiment, encoding the image according to the quantization step length can be understood as encoding once for each amount of data per quantization step length. Figure 2 and Figure 3For example, the distances between the first area, the second area and the third area and the gaze point gradually increase, and the corresponding quantization steps are n1, n2 and n3 respectively, and n1<n2<n3. For the image of the first area, encoding is performed once for every n1 amount of data, for the image of the second area, encoding is performed once for every n2 amount of data, and for the image of the third area, encoding is performed once for every n3 amount of data.

[0048] Optionally, after encoding the image based on the multiple regions and the encoding information, the following step is further included: projecting the gaze point into the encoded image to obtain a target image.

[0049] In this embodiment, the image that the user is gazing at is encoded by region, and the target bit rate of the area close to the gaze point is larger, and the target bit rate of the area far from the gaze point is smaller, so that the output image is displayed clearly near the gaze point and blurred far from the gaze point, which is beneficial to the subsequent eye movement analysis and reduces the image transmission volume. After the image is encoded, the data transmission volume becomes lower, which makes the transmission more convenient and speeds up the data transmission speed. Therefore, the acquisition frame rate of the second acquisition device can be increased or the display frame rate of the image display device can be increased, so that the encoded image and eye tracking data can be transmitted to the back end (including computers or mobile phones and other terminals) in real time and synchronously. After the back end is processed by the eye tracking algorithm, the real-time gaze point is obtained, and it is integrated into the encoded image and displayed in real time on the terminal, thereby improving the overall data utilization rate. After increasing the acquisition frame rate of the second acquisition device or the display frame rate of the image display device, the image and gaze point that the user is gazing at can be synchronously superimposed, thereby improving the overall quality of the data. Exemplary, Figure 4 is an example diagram of projecting a gaze point onto an image in this embodiment, such as Figure 4 As shown, the subject wears a wearable eye tracking device (the figure takes analytical glasses as an example) to gaze, and the wearable eye tracking device is connected to a terminal with a display function, and the eye tracking data and the encoded environmental image are transmitted to the terminal. The terminal calculates the subject's gaze point through an eye tracking algorithm and then superimposes it on the environmental image.

[0050] The technical solution of this embodiment synchronously obtains the user's line of sight direction and / or gaze point information and the image the user is gazing at; divides the image into multiple areas based on the line of sight direction and / or gaze point information, and the multiple areas are at least two; encodes the image based on the multiple areas and coding information, and the coding information includes at least one of the following: overall target bit rate, quantization step size. The image processing method based on eye tracking provided by the embodiment of the present invention encodes the image based on the divided multiple areas and coding information to reduce the size of the environmental image transmission amount, thereby making the transmission more convenient and speeding up the data transmission speed. It also brings the possibility of subsequently increasing the acquisition frame rate of the image acquisition device or increasing the display frame rate of the image display device, and ultimately improves the accuracy of eye tracking.

[0051] Embodiment 2

[0052] Figure 5 is a schematic diagram of the structure of an image processing device based on eye tracking provided by Embodiment 2 of the present invention. Figure 5 As shown, the device comprises:

[0053] An acquisition module 510 is used to synchronously acquire the user's line of sight direction and / or gaze point information and the image the user is gazing at;

[0054] A region division module 520, configured to divide the image into a plurality of regions based on the sight direction and / or gaze point information, wherein the plurality of regions is at least two;

[0055] The image encoding module 530 is used to encode the image based on multiple regions and encoding information, where the encoding information includes at least one of the following: an overall target bit rate and a quantization step size.

[0056] Optionally, when the encoding information is an overall target bit rate, the image encoding module 530 is further configured to:

[0057] Determine sub-target bit rates for multiple regions according to the overall target bit rate;

[0058] The multiple regions are encoded respectively according to the sub-target coding rates of the multiple regions.

[0059] Optionally, the image encoding module 530 is further configured to:

[0060] Obtaining allocation ratios corresponding to the multiple regions, wherein the allocation ratios are inversely proportional to distances between the multiple regions and the gaze point;

[0061] The sub-target bit rates included in the multiple regions are determined according to the overall target bit rate and the allocation ratio.

[0062] Optionally, when the encoding information is a quantization step size, the image encoding module 530 is further configured to:

[0063] Obtaining quantization step sizes corresponding to the multiple regions, wherein the quantization step sizes are proportional to distances between the multiple regions and the fixation point;

[0064] The multiple regions are encoded respectively according to the quantization step sizes of the multiple regions.

[0065] Optionally, the region division module 520 is further configured to:

[0066] Mapping gaze direction and / or gaze point information into the image;

[0067] The image is divided into a plurality of areas with the gaze point as the center and the edges of the set figures as the boundary lines.

[0068] Optionally, the acquisition module 510 is further configured to:

[0069] Acquiring eye tracking data, wherein the eye tracking data includes at least one of the following: eye image, electromyographic biopotential data, and infrared scanning data;

[0070] The user's sight direction and / or gaze point information is determined based on the eye tracking data.

[0071] Optionally, it also includes: a projection module for

[0072] The gaze point is projected onto the encoded image to obtain a target image.

[0073] The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not described in detail in this embodiment, please refer to the methods provided by all the above embodiments of the present invention.

[0074] Embodiment 3

[0075] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0076] like Figure 6As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0077] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0078] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as an image processing method based on eye tracking.

[0079] In some embodiments, the image processing method based on eye tracking may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image processing method based on eye tracking described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the image processing method based on eye tracking in any other appropriate manner (e.g., by means of firmware).

[0080] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0081] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0082] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0083] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0084] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0085] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0086] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0087] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image processing method based on eye tracking, characterized in that: include: Synchronously obtaining the user's line of sight direction and / or gaze point information and the image the user is gazing at; Dividing the image into a plurality of regions based on the sight line direction and / or gaze point information, wherein the plurality of regions is at least two; The image is encoded based on the multiple regions and encoding information, where the encoding information includes at least one of the following: an overall target bit rate and a quantization step size.

2. The method according to claim 1, characterized in that: When the encoding information is an overall target bit rate, encoding the image based on the multiple regions and the encoding information includes: Determine the sub-target bit rates of the multiple regions respectively according to the overall target bit rate; The multiple regions are encoded respectively according to the sub-target coding rates of the multiple regions.

3. The method according to claim 2, characterized in that Determining the sub-target bit rates of the multiple regions respectively according to the overall target bit rate includes: Acquire allocation ratios corresponding to the multiple regions, wherein the allocation ratios are inversely proportional to distances between the multiple regions and the gaze point; The sub-target bit rates included in the multiple regions are determined according to the overall target bit rate and the allocation ratio.

4. The method according to claim 1, characterized in that: When the encoding information is a quantization step size, encoding the image based on the multiple regions and the encoding information includes: Acquire quantization step sizes corresponding to the multiple regions, wherein the quantization step sizes are proportional to distances between the multiple regions and the gaze point; The multiple regions are respectively encoded according to the quantization step sizes of the multiple regions.

5. The method according to claim 1, characterized in that Dividing the image into a plurality of regions based on the sight direction and / or gaze point information includes: Mapping the gaze direction and / or gaze point information into the image; The image is divided into a plurality of regions with the gaze point as the center and the edges of the set figures as the boundaries.

6. The method according to claim 1, characterized in that Obtain the user's gaze direction and / or gaze point information, including: Acquiring eye tracking data, wherein the eye tracking data includes at least one of the following: eye image, electromyographic biopotential data, and infrared scanning data; The user's sight direction and / or gaze point information is determined based on the eye tracking data.

7. The method according to claim 1, characterized in that After encoding the image based on the plurality of regions and the encoding information, the method further includes: The gaze point is projected onto the encoded image to obtain a target image.

8. An image processing device based on eye tracking, characterized in that: include: An acquisition module, used for synchronously acquiring the user's sight direction and / or gaze point information and the image the user is gazing at; A region division module, used for dividing the image into a plurality of regions based on the sight line direction and / or gaze point information, wherein the plurality of regions is at least two; The image encoding module is used to encode the image based on the multiple regions and encoding information, wherein the encoding information includes at least one of the following: an overall target bit rate and a quantization step size.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the eye-tracking-based image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image processing method based on eye tracking according to any one of claims 1 to 8 when executed.

Images