Image processing method and device and electronic equipment

Abstract

The invention discloses an image processing method and device and electronic equipment, and belongs to the technical field of image processing. The image processing method is applied to the head-mounted display device, the head-mounted display device comprises an infrared light source, an infrared camera, a motor and a light machine, the motor is used for adjusting the position of the light machine, and the relative position of the light machine and the infrared camera is kept unchanged; the image processing method comprises the steps of obtaining at least two frames of eyeball images of an optical machine at different positions through an infrared camera under the condition that an infrared light source is turned on; wherein each frame of eyeball image comprises at least one light spot, and the positions of the light spots in the eyeball images are at least partially different; according to the positions of the light spots in the eyeball images, fusing the eyeball images to obtain a target eyeball image; and executing a preset operation according to the target eyeball image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image processing method, device and electronic equipment. Background Art

[0002] Iris recognition or eye tracking both rely on the collected eye images. In order to obtain eye images with clear texture, infrared light sources are usually required during image collection. The presence of infrared light sources can easily lead to light spots in the collected eye images, thereby blocking the iris texture or eye, affecting the iris recognition effect or eye tracking effect. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an image processing method, device and electronic device, which can solve the technical problem in the related art that the iris recognition effect or eye tracking effect is poor due to light spot occlusion.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to a head-mounted display device, wherein the head-mounted display device includes an infrared light source, an infrared camera, a motor, and an optical machine, wherein the motor is used to adjust the position of the optical machine, and the relative positions of the optical machine and the infrared camera remain unchanged; the method includes:

[0005] When the infrared light source is turned on, at least two frames of eyeball images are obtained by the infrared camera at different positions of the optical machine; wherein each frame of the eyeball image contains at least one light spot, and the positions of the light spots in the eyeball images are at least partially different;

[0006] According to the positions of the light spots in the eyeball images, the eyeball images are fused to obtain the target eyeball image;

[0007] Perform preset operations based on the target eye image.

[0008] In a second aspect, an embodiment of the present application provides an image processing device, which is applied to a head-mounted display device, wherein the head-mounted display device includes an infrared light source, an infrared camera, a motor, and an optical machine, wherein the motor is used to adjust the position of the optical machine, and the relative positions of the optical machine and the infrared camera remain unchanged; the device includes:

[0009] An acquisition module is used to acquire at least two frames of eyeball images at different positions of the optical machine through an infrared camera when the infrared light source is turned on; wherein each frame of the eyeball image contains at least one light spot, and the positions of the light spots in each eyeball image are at least partially different;

[0010] A fusion module, used for fusing the eyeball images according to the positions of the light spots in the eyeball images to obtain a target eyeball image;

[0011] The execution module is used to execute preset operations according to the target eye image.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0013] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0014] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect.

[0015] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0016] In an embodiment of the present application, a head-mounted display device includes an infrared light source, an infrared camera, a motor and an optical machine. The motor is used to adjust the position of the optical machine. The relative positions of the optical machine and the infrared camera remain unchanged. When the infrared light source is turned on, at least two frames of eyeball images of the optical machine at different positions are obtained through the infrared camera; wherein each frame of the eyeball image contains at least one light spot, and the positions of the light spots in each eyeball image are at least partially different; according to the positions of the light spots in each eyeball image, each eyeball image is fused to obtain a target eyeball image; and a preset operation is performed according to the target eyeball image. In an embodiment of the present application, when the infrared light source is turned on, the position of the optical machine is adjusted by a motor, and at least two frames of images of the optical machine at different positions are collected through the infrared camera, so that the positions of the light spots in different frames of images change. By fusing multiple frames of images, the problem of light spot occlusion can be effectively solved. In this way, when iris recognition or eyeball tracking is performed based on the fused image, the iris recognition effect and the eyeball tracking effect can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of an eyeball image provided for related technology;

[0018] Figure 2 A schematic diagram of the structure of a head mounted display device provided in an embodiment of the present application;

[0019] Figure 3 A schematic diagram of the layout of an infrared lamp provided in an embodiment of the present application;

[0020] Figure 4 A flowchart of an image processing method provided in an embodiment of the present application;

[0021] Figure 5 A schematic diagram of a detection image provided in an embodiment of the present application;

[0022] Figure 6 A schematic diagram of a binary image provided in an embodiment of the present application;

[0023] Figure 7 A schematic diagram of another binary image provided in an embodiment of the present application;

[0024] Figure 8 A schematic diagram of another binary image provided in an embodiment of the present application;

[0025] Fig. 9 A schematic diagram of an eyeball image provided in an embodiment of the present application;

[0026] Fig.10 A schematic diagram of another eyeball image provided in an embodiment of the present application;

[0027] Fig.11 A schematic diagram of a target eyeball image provided in an embodiment of the present application;

[0028] Fig.12 A flowchart of another image processing method provided in an embodiment of the present application;

[0029] Fig.13 A flowchart of another image processing method provided in an embodiment of the present application;

[0030] Fig.14 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0031] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0032] Fig.16 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0034] The terms "first", "second", etc. in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are usually a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0035] As mentioned above, in order to use the head mounted display device to capture an eyeball image with as clear texture as possible, it is necessary to arrange an infrared light source in the head mounted display device to achieve image acquisition with the help of the infrared light source. Figure 1 As shown, due to the presence of the infrared light source, there are light spots in the collected eye image 100, and these light spots will block the iris area or the pupil area, thereby affecting the iris recognition effect or the eye tracking effect.

[0036] To this end, the embodiments of the present application provide an image processing method, device, and electronic device, which can solve the technical problem in the related art that the iris recognition effect or eye tracking effect is poor due to light spot occlusion.

[0037] The image processing method, device and electronic device provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0038] The image processing method provided in the embodiment of the present application can be applied to a head-mounted display device, which can include, for example, an augmented reality (AR) device, a virtual reality (VR) device, a mixed reality (MR) device, etc. The embodiment of the present application can be applied to scenes such as identity recognition and eye tracking.

[0039] like Figure 2 As shown, the head mounted display device 200 may include an infrared light source 201, an infrared camera 202, a motor 203, an optical machine 204 and a controller 205. The infrared light source 201, the infrared camera 202 and the motor 203 are respectively connected to the controller 205, and the motor 203 is also connected to the optical machine 204. The infrared camera 202 may be arranged at an edge of the optical machine 204, and the relative position of the infrared camera 202 and the optical machine 204 may remain unchanged, that is, the infrared camera 202 is stationary relative to the optical machine 204.

[0040] The controller 205 is used to control the working states of the infrared light source 201, the infrared camera 202 and the motor 203. For example, the controller 205 can control the lighting state of the infrared light source 201, the shooting state of the infrared camera 202, and the running state of the motor 203. The motor 203 can drive the optical machine 204 to move under the drive of the controller 205, so as to change the position of the optical machine 204. Since the relative positions of the optical machine 204 and the infrared camera 202 remain unchanged, the infrared camera 202 moves synchronously with the optical machine 204. When the human eye does not move, by changing the position of the infrared camera 202, an eyeball image with different light spot positions can be obtained, which provides a basis for subsequent image processing.

[0041] Exemplarily, the optical machine 204 may include components such as a waveguide lens, an optical machine module, and an optical lens. The waveguide lens is mainly used to guide the light beam into the user's eyes. The optical machine module mainly includes an optical machine body and a connecting part connected to the optical machine body, which is usually installed in the temple assembly and is used to control the optical machine body to project the image beam to the waveguide lens. The optical lens is mainly used to magnify the image and project it onto the retina to present an enlarged virtual object image.

[0042] The infrared light source 201 can be set on the side of the head mounted display device 200 close to the user, or on the side away from the user. In some embodiments, the infrared light source 201 can be an infrared lamp, and there can be multiple infrared lamps. The multiple infrared lamps can be evenly arranged, that is, the arc lengths between two adjacent infrared lamps are the same, or unevenly distributed, that is, the arc lengths between two adjacent infrared lamps are different, and can also be partially evenly distributed and partially unevenly distributed, as long as it can be ensured that a certain number of light spots can be included in the collected eyeball image when the human eye looks left, right, up or down, so that the iris texture in the collected eyeball image can be clearer.

[0043] Figure 3 An exemplary layout diagram of an infrared lamp is listed below. Figure 3 Taking the arrangement of 10 infrared lamps in a head-mounted display device as an example, which are numbered 1-10 respectively, a larger number of infrared lamps may be included in actual applications.

[0044] Figure 4 A flowchart of an image processing method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the image processing method may include the following steps:

[0045] S410: When the infrared light source is turned on, at least two frames of eyeball images at different positions of the optical machine are obtained by using an infrared camera.

[0046] Each frame of the eyeball image contains at least one light spot, and the positions of the light spots in each eyeball image are at least partially different.

[0047] S420: According to the positions of the light spots in the eyeball images, the eyeball images are fused to obtain a target eyeball image.

[0048] S430: Execute a preset operation according to the target eyeball image.

[0049] In an embodiment of the present application, a head-mounted display device includes an infrared light source, an infrared camera, a motor and an optical machine. The motor is used to adjust the position of the optical machine. The relative positions of the optical machine and the infrared camera remain unchanged. When the infrared light source is turned on, at least two frames of eyeball images of the optical machine at different positions are obtained through the infrared camera; wherein each frame of the eyeball image contains at least one light spot, and the positions of the light spots in each eyeball image are at least partially different; according to the positions of the light spots in each eyeball image, each eyeball image is fused to obtain a target eyeball image; and a preset operation is performed according to the target eyeball image. In an embodiment of the present application, when the infrared light source is turned on, the motor is used to adjust the position of the optical machine, and at least two frames of images of the optical machine at different positions are collected, so that the positions of the light spots in different frames of images change. By fusing multiple frames of images, the problem of light spot occlusion can be effectively solved. In this way, when iris recognition or eyeball tracking is performed based on the fused image, the iris recognition effect and the eyeball tracking effect can be improved.

[0050] The above steps are described in detail below:

[0051] In S410, illustratively, when the infrared light source includes a plurality of infrared lamps, turning on the infrared light source here may mean that all the infrared lamps are turned on, or some of the infrared lamps are turned on and some of the infrared lamps are turned off.

[0052] The states of the infrared light sources corresponding to the optical machine at different positions may be the same or different. For example, when all the infrared lights are turned on, the states of the infrared light sources corresponding to the optical machine at different positions may be the same, that is, when the optical machine is at different positions, all the infrared lights are turned on.

[0053] Exemplarily, when only some of the infrared lights are turned on, the states of the infrared lights corresponding to different positions of the optical machine may be different. For example, when the optical machine is in position 1, the infrared lights at odd-numbered positions are turned on, and when the optical machine is in position 2, the infrared lights at even-numbered positions are turned on.

[0054] Exemplarily, when the optical machine is in position 1, all infrared lamps are turned on, and when the optical machine is in position 2, some infrared lamps are turned on.

[0055] In the embodiment of the present application, when the infrared light source is turned on, the infrared camera can collect at least two frames of eye images at different positions of the optical machine. The at least two frames of eye images here can be continuous eye images, and the time interval between two adjacent frames of eye images is very short. In this way, multiple frames of images can be collected while ensuring that the eyes do not move, thereby improving the subsequent iris recognition effect or eye tracking effect.

[0056] For example, when the infrared light source is turned on and the optical machine is in the current position, the controller of the head mounted display device can control the infrared optical machine to collect a frame of eyeball image, and then the controller can use the motor to quickly move the optical machine 1-2 mm, and control the infrared camera to collect a frame of eyeball image after the position of the optical machine is changed, so as to obtain two frames of continuous eyeball image. In actual application, more frames of continuous eyeball image can also be collected while ensuring that the eyeball does not move.

[0057] Since the optical and mechanical positions corresponding to each frame of the eye image are different, the positions of the light spots in each frame of the eye image will also be partially or completely different, providing a basis for subsequent processing of the eye image.

[0058] In S420, since the position of the light spot in different eyeball images changes, the light spot can be reduced or even eliminated by fusing multiple frames of eyeball images, effectively solving the problem of the light spot blocking the iris texture or the eyeball.

[0059] The embodiment of the present application does not limit the specific fusion process. For example, an image fusion algorithm such as a Poisson fusion algorithm, a Laplace pyramid fusion algorithm, etc. can be used to fuse the eye images of each frame to obtain a target eye image without light spots.

[0060] Exemplarily, each frame of eyeball images may be fused in the following manner:

[0061] For each frame of the eyeball image, edge detection is performed on the eyeball image to obtain a detection image including the pupil edge and the iris edge;

[0062] Binarization is performed on the detection image to obtain a binary image;

[0063] Determine the light spot position within the target edge according to the binary image, where the target edge is at least one of the pupil edge and the iris edge;

[0064] According to the position of the light spot, the features of each eyeball image are complemented to obtain the target eyeball image.

[0065] For example, edge detection can be performed on each frame of eyeball image to obtain pupil edge and iris edge by edge detection algorithm. The edge detection algorithm can adopt, for example, Canny detection algorithm, Sobel detection algorithm, Prewitt detection algorithm, etc.

[0066] Figure 5 Schematic diagram of a detection image provided by an embodiment of the present application, the detection image 500 includes a detected pupil edge 501 and an iris edge 502, wherein the white portion is a light spot 503, Figure 5 It can be seen that the inside of the pupil and between the pupil edge 501 and the iris edge 502 contain a certain number of light spots, which block the iris and the pupil.

[0067] In order to determine the effective light spot, the detection image can be binarized to obtain a binarized image. The threshold corresponding to the binarization can be set according to the actual situation. In some embodiments, the binarization threshold can be set to 200, that is, the pixel points with gray values ​​greater than 200 can be represented by 1, and the pixel points with gray values ​​less than 200 can be represented by 0. In this way, Figure 6 A binarized image 600 is shown.

[0068] Figure 7 An exemplary embodiment includes Figure 5 Schematic diagram of a binary image 700 of a detected pupil edge 501 and an iris edge 502, Figure 7 It can be determined which light spots belong to the blocking light spots, for example, the light spots located within the target edge can be called blocking light spots, and the light spots located outside the target edge can be called non-blocking light spots. The target edge is at least one of the pupil edge and the iris edge.

[0069] For example, in an iris recognition scenario, the target edge may be the pupil edge and the iris edge, that is, the light spot between the pupil edge and the iris edge is an occluded light spot, and the light spot outside the iris edge or inside the pupil edge is a non-occluded light spot.

[0070] For another example, in an eye tracking scenario, the target edge may be a pupil edge, that is, a light spot located inside the pupil edge is an occluded light spot, and a light spot located outside the pupil edge is a non-occluded light spot.

[0071] For another example, in a mixed scenario of iris recognition and eye tracking, the target edge may be the iris edge, that is, the light spot inside the iris edge is an occluded light spot, and the light spot outside the iris edge is a non-occluded light spot.

[0072] Figure 8 A schematic diagram of a binary image 800 applied to an iris recognition scenario is exemplified, that is, the light spot 503 located between the pupil edge and the iris edge is an occluded light spot.

[0073] Since the positions of the light spots in different eyeball images are different, the features of the eyeball images can be complemented according to the positions of the light spots to obtain an eyeball image without light spots.

[0074] Figure 9-10 Two frames of eyeball images are exemplified, namely, eyeball image 900 and eyeball image 1000. Taking the application in iris recognition scenario as an example, the two frames of images are processed according to the above process and feature complementation is performed to obtain Fig.11 The iris image 1100 is shown without light spots. Fig.11 It can be seen that through feature complementarity, the light spot between the edge of the pupil and the edge of the iris can be eliminated, effectively solving the problem of the light spot blocking the iris.

[0075] The embodiment of the present application obtains multiple frames of eye images with different light spot positions and effectively solves the light spot occlusion problem by fusing the multiple frames of images, thereby improving the subsequent iris recognition effect or eye tracking effect.

[0076] In S430, after the light spot is eliminated, a preset operation can be performed according to the target eye image after the light spot is eliminated. The preset operation here can be an identity recognition operation, an eye tracking operation, etc.

[0077] Taking the iris recognition operation as an example, illustratively, the iris features of the target eyeball image can be extracted according to the iris recognition process, and identity recognition can be performed based on the iris features.

[0078] Fig.12 A flowchart of another image processing method provided in an embodiment of the present application, Fig.12 For example, the infrared light source includes N infrared lamps, where N is an integer greater than 1. Fig.12 and Figure 4 The difference is that Figure 4 The S410 in can be refined into Fig.12 The S1200 in the.

[0079] S1200: When all infrared lights are turned on, obtain at least two frames of eyeball images at different positions of the optical machine through an infrared camera.

[0080] For example, when all infrared lights are on, the controller can control the infrared camera to capture one frame of image while the optical machine position remains unchanged. When all infrared lights remain fully lit, the controller can control the optical machine to move 1-2 mm quickly through the motor and control the infrared camera to capture the next frame of eyeball image. In this way, two consecutive frames of eyeball images with different light spot positions can be obtained.

[0081] In the embodiment of the present application, when all the infrared lamps are lit and the status remains unchanged, by moving the position of the optical machine, multiple frames of eye images with different light spot positions can be obtained, which provides a basis for subsequent elimination of the light spot.

[0082] Fig.13A flowchart of another image processing method provided in an embodiment of the present application, Fig.13 Taking the example that the infrared light source includes N infrared lamps, N is an integer greater than 1, and the different positions include a first position and a second position, the first position and the second position are different positions. Fig.13 and Figure 4 The difference is that Figure 4 The S410 in can be refined into Fig.13 S1300-S1310 in.

[0083] S1300: When the first infrared light is turned on, obtain a first eyeball image of the optical machine at the first position through an infrared camera.

[0084] The first infrared lamp is a part of the N infrared lamps.

[0085] S1310. When the second infrared light is turned on, obtain a second eyeball image of the optical machine at the second position through the infrared camera.

[0086] The second infrared lamp is a part of the N infrared lamps, and the second infrared lamp is at least partially different from the first infrared lamp.

[0087] The embodiment of the present application does not specifically limit the positions of the first infrared lamp and the second infrared lamp, as long as it can be ensured that the first infrared lamp and the second infrared lamp are at least partially different.

[0088] by Figure 3 Taking the infrared lamps shown in FIG. 1 as an example, in some embodiments, the first infrared lamp may be an infrared lamp numbered 1-5, and the second infrared lamp may be an infrared lamp numbered 6-10. In some embodiments, the first infrared lamp may also be an infrared lamp numbered odd, and the second infrared lamp may also be an infrared lamp numbered even. In some embodiments, the number of the first infrared lamp and the second infrared lamp may also be different.

[0089] In some embodiments, the lighting status of the infrared light can also be associated with the frame number of the eye image. For example, when the first eye image is an odd-numbered frame image, the first infrared light is the infrared light located at an odd-numbered position among the N infrared lights; when the second eye image is an even-numbered frame image, the second infrared light is the infrared light located at an even-numbered position among the N infrared lights.

[0090] Exemplarily, when the first eyeball image is the first frame, the first infrared lamp may be an infrared lamp located at an odd position, that is, an infrared lamp numbered 1, 3, 5, 7, and 9. That is, when the infrared lamps numbered 1, 3, 5, 7, and 9 are turned on and the infrared lamps numbered 2, 4, 6, 8, and 10 are turned off, the controller may control the infrared camera to collect a frame of the eyeball image at the first position.

[0091] Exemplarily, when the second eyeball image is the second frame, the second infrared lamp may be an infrared lamp located at an even-numbered position, that is, an infrared lamp numbered 2, 4, 6, 8, and 10. That is, when the infrared lamps numbered 2, 4, 6, 8, and 10 are turned on and the infrared lamps numbered 1, 3, 5, 7, and 9 are turned off, the controller may control the infrared camera to collect a frame of the eyeball image at the second position.

[0092] Of course, when the first eye image is the first frame, the first infrared light may be an infrared light numbered 2, 4, 6, 8, or 10; and when the second eye image is the second frame, the second infrared light may be an infrared light numbered 1, 3, 5, 7, or 9.

[0093] The embodiment of the present application can dynamically change the position of the light spot in different frame images according to the lighting state of the infrared lamp and the position of the optical machine, thereby improving the flexibility of the method.

[0094] The embodiments of the present application adjust the lighting state of the infrared lamp and / or change the position of the optical machine, so that multiple frames of eye images with different light spot positions can be collected, and the occlusion of the eye by the light spot is effectively reduced through image fusion, thereby improving the subsequent iris recognition effect and eye tracking effect.

[0095] It should be noted that the image processing method provided in the embodiment of the present application can be executed by an image processing device or a processing module in the image processing device for executing the image processing method. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing device executing the image processing method as an example.

[0096] Fig.14 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application.

[0097] like Fig.14 As shown, the image processing device 1400 may include:

[0098] The acquisition module 1401 is used to acquire at least two frames of eyeball images at different positions of the optical machine through an infrared camera when the infrared light source is turned on; wherein each frame of the eyeball image contains at least one light spot, and the positions of the light spots in the eyeball images are at least partially different;

[0099] A fusion module 1402 is used to fuse the eye images according to the positions of the light spots in the eye images to obtain a target eye image;

[0100] The execution module 1403 is used to execute a preset operation according to the target eye image.

[0101] In an embodiment of the present application, a head-mounted display device includes an infrared light source, an infrared camera, a motor and an optical machine. The motor is used to adjust the position of the optical machine. The relative positions of the optical machine and the infrared camera remain unchanged. When the infrared light source is turned on, at least two frames of eyeball images of the optical machine at different positions are obtained through the infrared camera; wherein each frame of the eyeball image contains at least one light spot, and the positions of the light spots in each eyeball image are at least partially different; according to the positions of the light spots in each eyeball image, each eyeball image is fused to obtain a target eyeball image; and a preset operation is performed according to the target eyeball image. In an embodiment of the present application, when the infrared light source is turned on, the motor is used to adjust the position of the optical machine, and at least two frames of images of the optical machine at different positions are collected, so that the positions of the light spots in different frames of images change. By fusing multiple frames of images, the problem of light spot occlusion can be effectively solved. In this way, when iris recognition or eyeball tracking is performed based on the fused image, the iris recognition effect and the eyeball tracking effect can be improved.

[0102] In some possible implementations of the embodiments of the present application, the infrared light source includes N infrared lamps, where N is an integer greater than 1;

[0103] The acquisition module 1401 is specifically used for:

[0104] When all infrared lights are turned on, at least two frames of eyeball images at different positions of the optical machine are obtained through the infrared camera.

[0105] In some possible implementations of the embodiments of the present application, the infrared light source includes N infrared lamps, N is an integer greater than 1, and the different positions include a first position and a second position;

[0106] The acquisition module 1401 is specifically used for:

[0107] When the first infrared light is turned on, a first eyeball image of the optical machine at the first position is acquired by the infrared camera, and the first infrared light is a part of the N infrared lights;

[0108] When the second infrared light is turned on, a second eyeball image of the optical machine at the second position is acquired by the infrared camera, the second infrared light is a part of the N infrared lights, and the second infrared light is at least partially different from the first infrared light.

[0109] In some possible implementations of the embodiments of the present application, when the first eyeball image is an odd-numbered frame image, the first infrared lamp is an infrared lamp located at an odd-numbered position among the N infrared lamps;

[0110] When the second eyeball image is an even-numbered frame image, the second infrared lamp is an infrared lamp located at an even-numbered position among the N infrared lamps.

[0111] In some possible implementations of the embodiment of the present application, the fusion module 1402 is specifically configured to:

[0112] For each frame of the eyeball image, edge detection is performed on the eyeball image to obtain a detection image including the pupil edge and the iris edge;

[0113] Binarization is performed on the detection image to obtain a binary image;

[0114] Determine the light spot position within the target edge according to the binary image, where the target edge is at least one of the pupil edge and the iris edge;

[0115] According to the position of the light spot, the features of each eyeball image are complemented to obtain the target eyeball image.

[0116] The embodiments of the present application adjust the lighting state of the infrared lamp and / or change the position of the optical machine, so that multiple frames of eye images with different light spot positions can be collected, and the occlusion of the eye by the light spot is effectively reduced through image fusion, thereby improving the subsequent iris recognition effect and eye tracking effect.

[0117] The image processing device in the embodiment of the present application can be a device, or a component in an electronic device, such as an integrated circuit or a chip. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a user computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not make specific limitations.

[0118] The electronic device in the embodiment of the present application may be a terminal with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0119] The image processing device provided in the embodiment of the present application can achieve Figures 1 to 13 The various processes in the image processing method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described here.

[0120] like Fig.15 As shown, an embodiment of the present application further provides an electronic device 1500, including a processor 1501 and a memory 1502, wherein the memory 1502 stores programs or instructions that can be executed on the processor 1501, and when the program or instructions are executed by the processor 1501, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0121] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile terminals and non-mobile terminals.

[0122] Fig.16 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0123] The electronic device 1600 includes but is not limited to: a radio frequency unit 1601, a network module 1602, an audio output unit 1603, an input unit 1604, a sensor 1605, a display unit 1606, a user input unit 1607, an interface unit 1608, a memory 1609, and a processor 1610 and other components.

[0124] Those skilled in the art will appreciate that the electronic device 1600 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1610 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.16 The structure of the electronic device 1600 shown in the figure does not constitute a limitation on the electronic device 1600. The electronic device 1600 may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0125] The processor 1610 is used to obtain at least two frames of eyeball images at different positions of the optical machine through the infrared camera when the infrared light source is turned on; each frame of the eyeball image contains at least one light spot, and the positions of the light spots in each eyeball image are at least partially different; according to the positions of the light spots in each eyeball image, the eyeball images are fused to obtain the target eyeball image;

[0126] Perform preset operations based on the target eye image.

[0127] In an embodiment of the present application, a head-mounted display device includes an infrared light source, an infrared camera, a motor and an optical machine. The motor is used to adjust the position of the optical machine. The relative positions of the optical machine and the infrared camera remain unchanged. When the infrared light source is turned on, at least two frames of eyeball images of the optical machine at different positions are obtained through the infrared camera; wherein each frame of the eyeball image contains at least one light spot, and the positions of the light spots in each eyeball image are at least partially different; according to the positions of the light spots in each eyeball image, each eyeball image is fused to obtain a target eyeball image; and a preset operation is performed according to the target eyeball image. In an embodiment of the present application, when the infrared light source is turned on, the motor is used to adjust the position of the optical machine, and at least two frames of images of the optical machine at different positions are collected, so that the positions of the light spots in different frames of images change. By fusing multiple frames of images, the problem of light spot occlusion can be effectively solved. In this way, when iris recognition or eyeball tracking is performed based on the fused image, the iris recognition effect and the eyeball tracking effect can be improved.

[0128] In some possible implementations of the embodiments of the present application, the infrared light source includes N infrared lamps, where N is an integer greater than 1;

[0129] The processor 1610 is specifically configured to:

[0130] When all infrared lights are turned on, at least two frames of eyeball images at different positions of the optical machine are obtained through the infrared camera.

[0131] In some possible implementations of the embodiments of the present application, the infrared light source includes N infrared lamps, N is an integer greater than 1, and the different positions include a first position and a second position;

[0132] The processor 1610 is specifically configured to:

[0133] When the first infrared light is turned on, a first eyeball image of the optical machine at the first position is acquired by the infrared camera, and the first infrared light is a part of the N infrared lights;

[0134] When the second infrared light is turned on, a second eyeball image of the optical machine at the second position is acquired by the infrared camera, the second infrared light is a part of the N infrared lights, and the second infrared light is at least partially different from the first infrared light.

[0135] In some possible implementations of the embodiments of the present application, when the first eyeball image is an odd-numbered frame image, the first infrared lamp is an infrared lamp located at an odd-numbered position among the N infrared lamps;

[0136] When the second eyeball image is an even-numbered frame image, the second infrared lamp is an infrared lamp located at an even-numbered position among the N infrared lamps.

[0137] In some possible implementations of the embodiments of the present application, the processor 1610 is specifically configured to:

[0138] For each frame of the eyeball image, edge detection is performed on the eyeball image to obtain a detection image including the pupil edge and the iris edge;

[0139] Binarization is performed on the detection image to obtain a binary image;

[0140] Determine the light spot position within the target edge according to the binary image, where the target edge is at least one of the pupil edge and the iris edge;

[0141] According to the position of the light spot, the features of each eyeball image are complemented to obtain the target eyeball image.

[0142] The embodiments of the present application adjust the lighting state of the infrared lamp and / or change the position of the optical machine, so that multiple frames of eye images with different light spot positions can be collected, and the occlusion of the eye by the light spot is effectively reduced through image fusion, thereby improving the subsequent iris recognition effect and eye tracking effect.

[0143] It should be understood that in the embodiment of the present application, the input unit 1604 may include a graphics processor (Graphics Processing Unit, GPU) 16041 and a microphone 16042, and the graphics processor 16041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1606 may include a display panel 16061, and the display panel 16061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1607 includes a touch panel 16071 and at least one of other input devices 16072. The touch panel 16071 is also called a touch screen. The touch panel 16071 may include two parts: a touch detection device and a touch controller. Other input devices 16072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0144] The memory 1609 can be used to store software programs and various data. The memory 1609 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1609 may include a volatile memory or a non-volatile memory, or the memory 1609 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1609 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0145] The processor 1610 may include one or more processing units; optionally, the processor 1610 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1610.

[0146] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0147] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0148] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0149] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0150] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0151] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0153] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: Applied to a head mounted display device, the head mounted display device comprises an infrared light source, an infrared camera, a motor and an optical machine, the motor is used to adjust the position of the optical machine, and the relative positions of the optical machine and the infrared camera remain unchanged; the method comprises: When the infrared light source is turned on, at least two frames of eyeball images of the optical machine at different positions are acquired by the infrared camera; wherein each frame of the eyeball image contains at least one light spot, and the positions of the light spots in the eyeball images are at least partially different; According to the positions of the light spots in the eyeball images, the eyeball images are fused to obtain a target eyeball image; A preset operation is performed according to the target eyeball image.

2. The method according to claim 1, characterized in that The infrared light source includes N infrared lamps, where N is an integer greater than 1; When the infrared light source is turned on, obtaining at least two frames of eyeball images of the optical machine at different positions by the infrared camera includes: When all the infrared lights are turned on, at least two frames of eyeball images of the optical machine at different positions are acquired through the infrared camera.

3. The method according to claim 1, characterized in that The infrared light source includes N infrared lamps, N is an integer greater than 1, and the different positions include a first position and a second position; When the infrared light source is turned on, obtaining at least two frames of eyeball images of the optical machine at different positions by the infrared camera includes: When a first infrared lamp is turned on, a first eyeball image of the optical machine at the first position is acquired by the infrared camera, wherein the first infrared lamp is a part of the N infrared lamps; When the second infrared light is turned on, the second eye image of the optical machine at the second position is acquired by the infrared camera, and the second infrared light is a part of the N infrared lights, and the second infrared light is at least partially different from the first infrared light.

4. The method according to claim 3, characterized in that When the first eyeball image is an odd-numbered frame image, the first infrared lamp is an infrared lamp located at an odd-numbered position among the N infrared lamps; When the second eyeball image is an even-numbered frame image, the second infrared lamp is an infrared lamp located at an even-numbered position among the N infrared lamps.

5. The method according to any one of claims 1 to 4, characterized in that: The step of fusing the eyeball images according to the positions of the light spots in the eyeball images to obtain the target eyeball image comprises: For each frame of the eyeball image, edge detection is performed on the eyeball image to obtain a detection image including a pupil edge and an iris edge; Performing binarization processing on the detection image to obtain a binarized image; Determine a light spot position within a target edge according to the binary image, wherein the target edge is at least one of the pupil edge and the iris edge; According to the light spot position, feature complementation is performed on each of the eyeball images to obtain a target eyeball image.

6. An image processing device, characterized in that: Applied to a head-mounted display device, the head-mounted display device comprises an infrared light source, an infrared camera, a motor and an optical machine, the motor is used to adjust the position of the optical machine, and the relative positions of the optical machine and the infrared camera remain unchanged; The device comprises: An acquisition module, configured to acquire, when the infrared light source is turned on, at least two frames of eyeball images of the optical machine at different positions through the infrared camera; wherein each frame of the eyeball image contains at least one light spot, and the positions of the light spots in the eyeball images are at least partially different; A fusion module, used for fusing the eyeball images according to the positions of the light spots in the eyeball images to obtain a target eyeball image; An execution module is used to execute a preset operation according to the target eye image.

7. The device according to claim 6, characterized in that The infrared light source includes N infrared lamps, where N is an integer greater than 1; The acquisition module is specifically used for: When all the infrared lights are turned on, at least two frames of eyeball images of the optical machine at different positions are acquired through the infrared camera.

8. The device according to claim 6, characterized in that The infrared light source includes N infrared lamps, N is an integer greater than 1, and the different positions include a first position and a second position; The acquisition module is specifically used for: When a first infrared lamp is turned on, a first eyeball image of the optical machine at the first position is acquired by the infrared camera, wherein the first infrared lamp is a part of the N infrared lamps; When the second infrared light is turned on, the second eye image of the optical machine at the second position is acquired by the infrared camera, and the second infrared light is a part of the N infrared lights, and the second infrared light is at least partially different from the first infrared light.

9. The device according to claim 8, characterized in that When the first eyeball image is an odd-numbered frame image, the first infrared lamp is an infrared lamp located at an odd-numbered position among the N infrared lamps; When the second eyeball image is an even-numbered frame image, the second infrared lamp is an infrared lamp located at an even-numbered position among the N infrared lamps.

10. The device according to any one of claims 6 to 9, characterized in that: The fusion module is specifically used for: For each frame of the eyeball image, edge detection is performed on the eyeball image to obtain a detection image including a pupil edge and an iris edge; Performing binarization processing on the detection image to obtain a binarized image; Determine a light spot position within a target edge according to the binary image, wherein the target edge is at least one of the pupil edge and the iris edge; According to the light spot position, feature complementation is performed on each of the eyeball images to obtain a target eyeball image.

11. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images