Focusing method and device based on eye movement monitoring information and electronic vehicle-mounted rearview mirror

Abstract

The embodiment of the invention discloses a focusing method and device based on eye movement monitoring information and an electronic vehicle-mounted rearview mirror. The focusing method based on the eye movement monitoring information comprises the following steps: receiving eye movement information of a driver acquired by eye movement detection equipment; determining a watching area of the driver on the display screen according to the eye movement information; and controlling the image sensor to focus based on the gazing area, so that the gazing area is clearly imaged. According to the embodiment of the invention, the image display effect on the display screen can be adjusted in real time along with the change of the attention of the driver, so that the driver can see a clear image, and the comfort and driving safety of the driver are improved.

Description

Technical Field

[0001] The present disclosure relates to artificial intelligence technology, and in particular to a focusing method, device and electronic vehicle rearview mirror based on eye movement monitoring information. Background Art

[0002] An indirect vision device is a device used by the driver to observe the adjacent traffic area of ​​the vehicle that cannot be observed by the direct field of vision. Take the electronic rearview mirror as an example. It is a new type of rearview mirror that can replace the traditional optical rearview mirror. It adopts a combined structure of a camera and a display device, which can change the field of view limitation of the traditional optical rearview mirror caused by factors such as the curvature, shape, size and angle of the mirror surface, making the entire field of view wider. However, the electronic rearview mirror is based on a camera and a display device, and is limited by the principle of optical imaging. The range of clear imaging can only be fixed within a certain range. Blurred images outside the depth of field can easily cause safety problems, and can also easily cause visual fatigue, causing the driver to feel dizzy and other discomfort. Summary of the invention

[0003] In view of the above problems in the prior art, the embodiments of the present disclosure provide a focusing method, device and electronic vehicle rearview mirror based on eye movement monitoring information. The embodiments of the present disclosure can adjust the image display effect on the display screen in real time as the driver's attention changes, so that the driver can see a clear image, thereby improving the driver's comfort and driving safety.

[0004] To achieve the above-mentioned object, the present disclosure provides a focusing method based on eye movement monitoring information in a first aspect, comprising:

[0005] Receiving the driver's eye movement information acquired by the eye movement detection device;

[0006] Determine the driver's gaze area on the display screen based on eye movement information;

[0007] The image sensor is controlled to focus based on the gaze area so that the gaze area is clearly imaged.

[0008] As a possible implementation manner of the first aspect, receiving eye movement information of a driver acquired by an eye movement detection device includes:

[0009] Collecting light signal information of a local spatial area and converting it into a pulse signal according to the light signal information; the light signal information includes optical flow information reflecting eye movement;

[0010] In response to the pulse signal, the pulse signal is input into an eye tracking model, and the eye tracking model outputs eye movement information corresponding to the pulse signal; the eye tracking model is generated based on a pulse neural network.

[0011] As a possible implementation of the first aspect, an eye movement detection device and a display screen are arranged in an electronic vehicle rearview mirror; and an image sensor is deployed at the rear, side or bottom of the vehicle.

[0012] As a possible implementation manner of the first aspect, the eye movement information includes image information of the driver's eyes and corneal reflection points.

[0013] As a possible implementation of the first aspect, determining the driver's gaze area on the display screen according to the eye movement information includes:

[0014] The image information of the driver's eyes and corneal reflection points is input into a pre-trained machine learning model, and the machine learning model outputs the driver's gaze area on the display screen.

[0015] As a possible implementation manner of the first aspect, controlling the image sensor to focus includes:

[0016] Focusing is done by adjusting the lens position of the image sensor.

[0017] As a possible implementation manner of the first aspect, a manner of adjusting the lens position of the image sensor for focusing includes at least one of a contrast detection method, a phase difference detection method, and a hybrid focus mode.

[0018] As a possible implementation manner of the first aspect, after controlling the image sensor to focus based on the gaze area so that the gaze area is clearly imaged, the method further includes:

[0019] Controls the depth of field range of the image shown in the monitor.

[0020] As a possible implementation manner of the first aspect, controlling a depth of field range of an image displayed on a display screen includes:

[0021] By adjusting the image sensor's lens aperture, the depth of field range of the image is controlled.

[0022] A second aspect of the present disclosure provides a focusing device based on eye movement monitoring information, comprising:

[0023] A receiving unit, used to: receive the driver's eye movement information acquired by the eye movement detection device;

[0024] A processing unit, configured to: determine a driver's gaze area on the display screen based on the eye movement information;

[0025] The adjustment unit is used to control the image sensor to focus based on the gaze area so that the gaze area is clearly imaged.

[0026] As a possible implementation manner of the second aspect, the receiving unit is configured to:

[0027] Collecting light signal information of a local spatial area and converting it into a pulse signal according to the light signal information; the light signal information includes optical flow information reflecting eye movement;

[0028] In response to the pulse signal, the pulse signal is input into an eye tracking model, and the eye tracking model outputs eye movement information corresponding to the pulse signal; the eye tracking model is generated based on a pulse neural network.

[0029] As a possible implementation of the second aspect, an eye movement detection device and a display screen are arranged in an electronic vehicle rearview mirror; and an image sensor is deployed at the rear, side or bottom of the vehicle.

[0030] As a possible implementation manner of the second aspect, the eye movement information includes image information of the driver's eyes and corneal reflection points.

[0031] As a possible implementation manner of the second aspect, the processing unit is configured to:

[0032] The image information of the driver's eyes and corneal reflection points is input into a pre-trained machine learning model, and the machine learning model outputs the driver's gaze area on the display screen.

[0033] As a possible implementation manner of the second aspect, the adjusting unit controls the image sensor to focus, including:

[0034] Focusing is done by adjusting the lens position of the image sensor.

[0035] As a possible implementation manner of the second aspect, the manner in which the adjustment unit adjusts the lens position of the image sensor for focusing includes at least one of a contrast detection method, a phase difference detection method, and a hybrid focus mode.

[0036] As a possible implementation manner of the second aspect, the apparatus further includes a control unit, where the control unit is configured to:

[0037] The image sensor is controlled to focus based on the gaze area, so that after the gaze area is clearly imaged, the depth of field range of the image displayed on the display screen is controlled.

[0038] As a possible implementation manner of the second aspect, the control unit is configured to:

[0039] By adjusting the image sensor's lens aperture, the depth of field range of the image is controlled.

[0040] A third aspect of the present disclosure provides an electronic vehicle-mounted rearview mirror, which includes an eye movement detection device, a display screen, a processor, and a plurality of image sensors deployed at the rear, side, and bottom of a vehicle, wherein:

[0041] The eye movement detection device is arranged at a position adapted to the display screen, and is used to obtain the driver's eye movement information and send the eye movement information to the processor;

[0042] The processor is connected to the eye movement detection device to receive the eye movement information; and according to the eye movement information, determines the driver's gaze area on the display screen of the electronic vehicle rearview mirror;

[0043] The processor is also connected to the image sensor and is used to control the image sensor to focus based on the gaze area so that the gaze area is clearly imaged.

[0044] A fourth aspect of the present disclosure provides a vehicle, comprising an electronic vehicle-mounted rearview mirror as described in any one of the third aspects above, the electronic vehicle-mounted rearview mirror serving as at least one of a left exterior rearview mirror, a right exterior rearview mirror and an interior center rearview mirror.

[0045] A fifth aspect of the present disclosure provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor, and further comprising the focusing device according to any one of the second aspects above;

[0046] Memory for storing computer-executable instructions;

[0047] The processor is used to execute the computer-executable instructions stored in the memory to control the focusing device based on eye movement monitoring information to implement the method described in any one of the first aspects above.

[0048] A sixth aspect of the present disclosure provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the method described in any one of the first aspects above is implemented.

[0049] A seventh aspect of the present disclosure provides a computer program product, including a computer program, which implements any method described in the first aspect when executed by a processor.

[0050] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0052] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0053] Figure 1 A schematic diagram of the human field of vision.

[0054] Figure 2The present invention is a schematic diagram of the structure of an electronic vehicle rearview mirror according to an embodiment of the focusing method based on eye movement monitoring information disclosed herein.

[0055] Figure 3 The present invention is a flowchart of an embodiment of a focusing method based on eye movement monitoring information.

[0056] Figure 4 This is a schematic diagram of eye tracking according to an embodiment of the focusing method based on eye movement monitoring information disclosed in the present invention.

[0057] Figure 5 This is a schematic diagram of eye tracking according to an embodiment of the focusing method based on eye movement monitoring information disclosed in the present invention.

[0058] Figure 6 This is a schematic diagram of eye tracking according to an embodiment of the focusing method based on eye movement monitoring information disclosed in the present invention.

[0059] Figure 7 The figure is a focusing schematic diagram of an embodiment of a focusing method based on eye movement monitoring information disclosed in the present invention.

[0060] Figure 8 The figure is a focusing schematic diagram of an embodiment of a focusing method based on eye movement monitoring information disclosed in the present invention.

[0061] Fig. 9 The figure is a focusing schematic diagram of an embodiment of a focusing method based on eye movement monitoring information disclosed in the present invention.

[0062] Fig.10 The present invention is a schematic diagram of the imaging principle of the lens of an image sensor according to an embodiment of the focusing method based on eye movement monitoring information disclosed herein.

[0063] Fig.11 The present invention is a schematic diagram of the imaging principle of the lens of an image sensor according to an embodiment of the focusing method based on eye movement monitoring information disclosed herein.

[0064] Fig.12 The present invention is a schematic diagram of adjusting the depth of field effect of an image sensor according to an embodiment of the focusing method based on eye movement monitoring information disclosed herein.

[0065] Fig.13 The present invention is a schematic diagram of adjusting the depth of field effect of an image sensor according to an embodiment of the focusing method based on eye movement monitoring information disclosed herein.

[0066] Fig.14 It is a structural schematic diagram of an embodiment of a focusing device based on eye movement monitoring information disclosed in the present invention;

[0067] Fig.15 It is a structural schematic diagram of an embodiment of a focusing device based on eye movement monitoring information disclosed in the present invention;

[0068] Fig.16A block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0069] An indirect field of view device is a device that obtains a specified field of view through a system composed of a camera and a monitor. Taking the on-board electronic rearview mirror as an example, since its camera area is small and it is less affected by the environment, it can also eliminate environmental influences through hardware and software technologies such as infrared sensing function and enhanced image contrast, so it can clearly present the image outside the car on the display screen inside the car. In addition, the electronic rearview mirror is also suitable for night environments. When the light is very poor, the road conditions can still clearly display the image, improve the visual effect at night, and enhance travel safety in various complex environments. Furthermore, since the external equipment of the electronic rearview mirror is only the sensor device of the camera, it can achieve a smaller size, further reduce wind resistance and wind noise, and even reduce the energy consumption of the entire vehicle. In addition, this sensor device can also record the surrounding vehicle conditions and driving environment, and may also become evidence for traffic liability determination.

[0070] However, the existing electronic rearview mirror is limited by the principle of optical imaging, and its clear imaging range can only be fixed within a certain range. Usually, the distance range in front of and behind the object measured by the front edge of the camera lens or other imager that can obtain a clear image is called depth of field. Generally, the camera lens must be focused before it can be taken. In theory, only the part that is accurately focused is clear in the photo, and the scenery in front of and behind the focus will appear blurry because it is out of focus. However, based on factors such as the lens and shooting distance, there will still be a distance of scenery in front of and behind the focus that can be clearly displayed and will not fall into the blurred area. This clear range is called depth of field. Depth of field refers to the acceptable clear area before and after the focus, that is, the clear range of the scene in the captured image.

[0071] When driving, the driver needs to observe the entire captured image in the rearview mirror, and may also pay attention to images outside the depth of field. Blurred images outside the depth of field of existing electronic rearview mirrors can easily cause safety problems and also cause discomfort to the driver. The disclosed embodiment is based on the principle of human eye vision and camera imaging principle, and uses eye tracking technology to capture eye movement information. According to the eye movement information, the human eye's attention area can be calculated, and then the autofocus system is adjusted to make the attention area clearly imaged, thereby effectively solving the above technical problems in the prior art.

[0072] Visual perception begins when light reflected from an object or scene enters the eye through the cornea, pupil, and lens. The 1% fixation area within the human eye's field of vision contains 50% of the visual information transmitted to the brain. Figure 1Schematic diagram of the human visual field shown. The main area F (fovea area) in the focus corresponds to the human gaze focus area. This area only accounts for 1% of the entire visual field, but the information recorded by this area contains 50% of the effective visual information transmitted to the brain through the optic nerve. Therefore, the embodiment of the present disclosure uses an eye tracker to collect the driver's eye movement information, and the driver's gaze area on the display screen of the electronic vehicle rearview mirror can be obtained based on the eye movement information. The gaze area is also the area of ​​human eye attention. Based on this area, the image sensor of the electronic vehicle rearview mirror is controlled to focus, so that the gaze area is clearly imaged, which can enhance the driver's visual experience and thus enhance driving safety.

[0073] Figure 2 The present invention is a schematic diagram of the structure of an electronic vehicle rearview mirror according to an embodiment of the focusing method based on eye movement monitoring information disclosed herein. Figure 2 The meanings of the numbers in are as follows: number 1 represents the eye movement detection device; number 2 represents the image sensor; number 3 represents the display screen; number 4 represents the gaze area of ​​the human eye.

[0074] The focusing method based on eye movement monitoring information provided by the embodiments of the present disclosure can be applied to various indirect vision devices, including electronic vehicle rearview mirrors. Figure 2 As shown, an eye movement detection device 1, a processor, an image sensor 2 and a display screen 4 can be set in the electronic vehicle rearview mirror. Among them, the image sensor is a tool for obtaining external information such as the rear, side and bottom of the car. The image sensor is a device that converts an optical image into an electronic signal, including a CCD (Charge-coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. Among them, the CMOS image sensor adopts a pulse modulation method, and realizes the continuous recording and digital expression of the light process by encoding the pulse characteristic value in the time unit corresponding to each sampling moment in the unit time. Based on the encoding information of the pulse signal in the unit time, it can be used to estimate the light intensity at any time, so as to obtain the image at any time, thereby realizing continuous imaging, having high time sensitivity, high-frequency expression ability and super strong expression ability for high-speed motion process, can effectively record high-speed motion, can capture the motion details in the observation scene, and can support the needs of high-precision detection.

[0075] See also Figure 2, wherein the eye movement detection device obtains the eye movement information of the human eye based on the eye tracking technology. The eye tracking technology locates the pupil position through image processing technology, and calculates the position of the eye gaze or gaze through the eye tracking algorithm on the processor, and determines the subject's gaze area on the display screen in real time. The processor controls the focus of the image sensor based on the gaze area, so that the gaze area is clearly imaged, and the image information captured by the image sensor after accurate focus is displayed on the display screen.

[0076] Eye movement detection equipment includes an eye tracker. In one example, the eye tracker includes an eye movement sensor, an infrared transmitter, and a processor. The infrared transmitter emits near-infrared light to the eyes, and the eye movement sensor collects high-definition images of the eyes and near-infrared reflection points. Finally, the processor determines the subject's eye position and gaze direction through image recognition, machine learning, and mathematical model algorithms, and then obtains the subject's gaze area on the screen.

[0077] Figure 3 FIG. 1 is a flow chart of an embodiment of the focusing method based on eye movement monitoring information disclosed in the present invention. Figure 3 As shown, the above method may specifically include:

[0078] Step S110, receiving the driver's eye movement information acquired by the eye movement detection device;

[0079] Step S120, determining the driver's gaze area on the display screen according to the eye movement information;

[0080] Step S130: Control the image sensor to focus based on the gaze area so that the gaze area is clearly imaged.

[0081] See also Figure 2 Taking the electronic vehicle rearview mirror as an example, in the electronic vehicle rearview mirror, the eye movement detection device 1, the image sensor 2 and the display screen 3 are all connected to the processor. Usually, the driver's head position is stable during driving. In this case, the driver's gaze direction can be determined based on the eye movement information collected by the eye movement detection device.

[0082] See also Figure 2 and Figure 3 In the disclosed embodiment, the eye movement detection device is used to collect the driver's eye movement information and transmit the eye movement information to the processor. In step S110, the processor receives the driver's eye movement information collected by the eye movement detection device. The eye movement information may specifically include the driver's eyes and the image information of the corneal reflection points where the infrared rays are reflected on the eyeballs. In step S120, the processor obtains the driver's gaze area 4 on the display screen based on the eye movement information through image recognition, machine learning and mathematical model algorithms. In step S130, based on the gaze area 4, the processor controls the image sensor 2 to focus so that the gaze area is clearly imaged.

[0083] Specifically, see Figure 2 and Figure 3 , the processor can adjust the position of the lens of the image sensor so that the driver's focus area is clearly imaged. After the image sensor 2 adjusts the focus, it can send the image data to the processor. The processor determines whether the focus is accurate based on the received image data. If the focus is not accurate enough, the processor determines the direction and position where the lens needs to move based on the image data, and controls the image sensor 2 to further focus to achieve more accurate focus. Figure 2 After the image sensor 2 has completed focusing, the image information of the gaze area that has been clearly imaged is sent to the display screen 3 for display.

[0084] In summary, the disclosed embodiments are based on eye tracking technology, and by capturing eye movement information, the human eye attention area can be calculated in real time. In one example, the processor can calculate the human eye attention area within 3ms, and adjust the autofocus system of the image sensor based on the human eye attention area, so that the human eye attention area is clearly imaged, and the image display effect on the display screen can be adjusted in real time as the driver's attention changes, so that the driver can see a clear image, improving the driver's comfort and driving safety.

[0085] In one embodiment, Figure 3 Step S110 in the embodiment of the present invention is to receive the eye movement information of the driver acquired by the eye movement detection device, including:

[0086] Collecting light signal information of a local spatial area and converting it into a pulse signal according to the light signal information; the light signal information includes optical flow information reflecting eye movement;

[0087] In response to the pulse signal, the pulse signal is input into an eye tracking model, and the eye tracking model outputs eye movement information corresponding to the pulse signal; the eye tracking model is generated based on a pulse neural network.

[0088] In visual perception, optical flow refers to the change in light pattern caused by the movement of an object in the field of vision, which can be perceived by the visual system and used to understand the movement of the object and its own position in space. The optical flow signal reflecting eye movement refers to the signal generated by the change in optical flow in the surrounding environment caused by eye movement. When the eyeball moves, the optical flow in the surrounding environment changes, and this change can be captured by the sensor and converted into a pulse signal. The pulse signal is used to reflect information such as the change in light intensity, the direction and speed of the light pattern. This information is used to reflect the movement state of the eyeball and the change in visual focus. Therefore, the optical flow signal reflecting eye movement is actually a signal that describes the change in optical flow caused by eye movement, which can be used to analyze eye movement behavior, identify fixation points, and track line of sight movement.

[0089] In the disclosed embodiment, a pulse vision sensor can be used to collect light signal information of a local spatial area, including optical flow information reflecting the driver's eye movement. The local spatial area includes the surrounding environment where the driver's eyes are located. The pulse vision sensor converts the collected optical flow signals reflecting the eye movement to generate a pulse signal, and represents the light intensity change process of each pixel collected over a period of time as a pulse signal stream. In this process, the pulse vision sensor receives the light signal from the eye and converts it into the form of a pulse signal. Such a conversion can better reflect the dynamic changes in the eye movement process and provide more accurate and detailed data for the subsequent output of eye movement information.

[0090] By setting up a pulse vision sensor, the optical flow signal collected by the pulse vision sensor is converted into a pulse signal, so there is no need for an imaging process. The pulse signal can better capture timing information and rapid changes, which helps to accurately capture eye movement behavior. The pulse signal is used as input and processed by the eye tracking model. The eye tracking model is generated based on a pulse neural network, which has characteristics more similar to biological nervous systems and can effectively process time correlation and the order of events. The neurons of the eye tracking model are excited according to the pattern and time characteristics of the input pulse, and then transmit and process information in the network. This processing method does not require the use of complex image processing algorithms to extract useful eye movement information from the captured image, and is closer to the working principle of the biological nervous system. The high temporal resolution and low energy consumption characteristics of the eye tracking model also help to achieve real-time tracking of eye movements with higher accuracy and lower power consumption.

[0091] Compared with traditional image-based visual sensors, pulse vision sensors are a type of sensor that works on a brain-like principle. They can directly convert photon streams into pulse streams, thereby preventing traditional image sensors from having to perform exposure imaging steps. Pulse vision sensors directly generate pulse signals, which are suitable for subsequent pulse neural network processing algorithms, greatly reducing perception delays while also capturing light signals at a higher frame rate. Furthermore, through real-time capture of eye movement information, the human eye's attention area can be calculated in real time, and then the image sensor's autofocus system can be adjusted based on the human eye's attention area, so that the human eye's attention area is clearly imaged. The image display effect on the display screen can be adjusted in real time as the driver's attention changes, thereby improving the driver's comfort and driving safety.

[0092] In one embodiment, the eye movement detection device and the display screen are arranged in the electronic vehicle rearview mirror; the image sensor is deployed at the rear, side or bottom of the vehicle.

[0093] The focusing method based on eye movement monitoring information provided by the embodiments of the present disclosure can be applied to various indirect vision devices, including automobile rearview mirrors, motorcycle rearview mirrors and camera-monitor systems (CMS). Indirect vision devices are mainly used to provide drivers with additional vision required during driving, helping to avoid blind spots and thus improve driving safety. These devices can be installed in various positions of motor vehicles, such as behind, to the side or below the vehicle, to provide different angles of vision, thereby helping the driver to better observe the environment around the vehicle.

[0094] In one implementation, the eye movement information includes image information of the driver's eyes and corneal reflection points.

[0095] In one example, an eye movement detection device can obtain the relative difference between the pupil center and the corneal reflection position, based on which the subject's gaze direction can be inferred. Figure 4 This is a schematic diagram of eye tracking according to an embodiment of the focusing method based on eye movement monitoring information disclosed in the present invention. Figure 4 In the figure, CAMERA means eye movement sensor, which is a camera installed in the eye movement detection device; PUPIL means pupil; IR (Infrared Radiation) LIGHT means infrared beam; CORNEAL REFLECTION means corneal reflection; GLINT means bright spot, also called corneal reflection point, which is the reflection point of light source on the cornea of ​​the eye.

[0096] like Figure 4 As shown in Figure 1, eye tracking technology is a "non-invasive" technology based on video oculographic (VOG). The basic principle is to aim a beam of light (such as near-infrared light) and an eye sensor at the subject's eyes, infer the direction of the subject's gaze through the light and back-end analysis, and the eye sensor records the interaction process. This video-based eye tracking can use one or more cameras to capture a series of eye images. In one example, in less than 3 milliseconds after the eye image is captured, the processor can calculate the coordinate position of the screen where the subject is looking. Specifically, the eye tracking software uses image processing algorithms to identify two key positions on each image sent by the eye tracking camera: the center of the pupil and the corneal reflection point.

[0097] like Figure 5As shown, the corneal reflection point is the reflection point on the cornea of ​​the light emitted by a fixed light source (such as an infrared transmitter). When the eye rotates, the position of the pupil center recorded on the eye movement sensor will change. When the head is stable, the positions of the eye movement sensor, infrared transmitter and corneal reflection point on the eye are relatively fixed. The subject's gaze area can be calculated based on the relative relationship between the positions of the corneal reflection point and the pupil.

[0098] Figure 6 The figure shows the eye image as seen by the camera when the eye looks ahead, then rotates to the participant's right, and then rotates to the participant's left. It can be seen that the corneal reflection point remains in approximately the same position in terms of camera pixel coordinates, while the center of the pupil moves. Therefore, if the eye is fixed in space and simply rotates around its own center, the gaze area can be determined by tracking the changes in the center of the pupil on the eye movement sensor alone.

[0099] In one example, the eye movement detection device may include a screen-type eye tracker. The screen-type eye tracker may be a rectangular device adsorbed under the screen, a portable telemetry type, or an all-in-one device combined with a screen. By algorithmically processing the eye image obtained by the eye movement sensor, the gaze direction vector of the eye facing the screen can be obtained. Combined with the displacement and rotation of the eye tracker relative to the screen, the position of the subject's gaze area relative to the screen coordinate system can be obtained through coordinate system conversion.

[0100] In one embodiment, determining the driver's gaze area on the display screen according to the eye movement information includes:

[0101] The image information of the driver's eyes and corneal reflection points is input into a pre-trained machine learning model, and the machine learning model outputs the driver's gaze area on the display screen.

[0102] In one example, an eye tracker may use a pulsed vision sensor to collect eye movement information. The pulsed vision sensor converts the captured light signal directly into a pulse signal without the need for imaging. This design can significantly reduce the system's perception delay, processing delay, and computational burden. In addition, a spiking neural network (SNN) can be used as a machine learning model for analyzing and processing eye movement information. A spiking neuron network is an artificial neural network model that simulates the working mode of neurons in a biological nervous system. Unlike traditional artificial neural network models (such as feedforward neural networks and recurrent neural networks), information transmission between neurons in a spiking neural network is achieved through pulse signals emitted by neurons. Using a spiking neural network to directly analyze the pulse signals captured by a spiking vision sensor is beneficial to reducing the computational burden and power consumption of the device and improving the operating efficiency of the device.

[0103] Pulse vision sensors can directly convert photon streams into pulse streams. Unlike traditional image sensors that require exposure and imaging steps, pulse vision sensors directly generate pulse signals, which are suitable for subsequent pulse neural network processing algorithms, greatly reducing perception delays and also achieving the capture of light signals at a higher frame rate.

[0104] In one example, an artificial neural network can be first trained using an eye movement information data set to obtain an initial model for eye movement tracking. The artificial neural network structure can be a deep neural network (DNN), a convolutional neural network (CNN), or a recurrent neural network (RNN). Then, based on the mapping relationship between the artificial neural network and the pulse neural network, the initial model is converted into a pulse neural network. In view of the characteristics of the pulse neural network, including pulse signal processing, timing characteristics, and synaptic weights, it is necessary to understand the differences and mapping relationships between the pulse neural network and the traditional artificial neural network. Based on the understanding of the mapping relationship, the initial model is converted accordingly. This process may involve steps such as adjusting weights and thresholds, processing timing information, etc., to ensure that similar eye movement tracking functions can be achieved in the SNN, and by adjusting the structure and parameters of the artificial neural network to better adapt to the processing and timing characteristics of the pulse signal. In addition, the conversion process may also include operations such as adjusting the activation function and changing the neuron connection method. The converted model has better pulse signal processing capabilities, can more effectively extract information from the pulse signal, and improves the accuracy of the model output results.

[0105] In one implementation, controlling the image sensor to focus includes: focusing by adjusting a lens position of the image sensor.

[0106] In one embodiment, the method of adjusting the lens position of the image sensor for focusing includes at least one of a contrast detection method, a phase difference detection method and a hybrid focus mode. The above focusing methods are respectively introduced below.

[0107] 1) Contrast detection method

[0108] Contrast refers to the difference between the grayscale values ​​of each pixel in an image from 0 to 255. The contrast of a successfully focused image is the greatest, the edges of objects in the image are clearer, and the colors are more vivid and bright.

[0109] Figure 7 The figure is a focusing schematic diagram of an embodiment of a focusing method based on eye movement monitoring information disclosed in the present invention. Figure 7 AF (Auto Focus) means auto focus. Figure 7 The "AF Comparison Range" in the image shows the grayscale of each pixel in a small range of the captured focus. For example, a small area of ​​64*64 pixels is captured in the focus area of ​​the image. The image contrast range corresponding to the lens moving to different positions is compared in the captured small area. When the contrast becomes the largest, the image in the focus area is the clearest.

[0110] The contrast detection method does not compare the contrast of the entire image, but compares the contrast of the image in a small captured area. In the prior art, the captured small area is usually the central area of ​​the entire image, which causes the edge area outside the depth of field to be blurred. When the driver focuses on the area outside the depth of field, he will feel uncomfortable. In the embodiment of the present disclosure, after obtaining the eye movement information, focusing is performed based on the gaze area, so that the area where the human eye is looking at is clearly imaged, thereby improving the image display effect and improving driving comfort.

[0111] See also Figure 7 After grabbing the grayscale of the pixels in the small area of ​​focus in the "AF comparison range", calculate the contrast of each pixel. Then rearrange the histogram to get the contrast histogram. Focusing is completed when the maximum value (MAX) is obtained in the contrast histogram.

[0112] See also Figure 8 Based on the above principle, a sensor can be placed at two locations with equal distance from the focal plane. The sensor can collect the contrast information of the image. Then the motor is controlled to move the focal plane, that is, to move the lens. Figure 8 C1 and C2 are the contrast change curves of the obtained image. The intersection of these two curves is the positive focus position. The contrast detection method is mainly based on photosensitive devices and image processors, does not require a separate focus auxiliary device, and has high focus accuracy.

[0113] 2) Phase detection method

[0114] The principle of phase detection is to split the light entering the lens into two, thus forming two images. Based on the difference between the focus positions of the two images, the direction and amount of lens movement required is calculated, such as calculating whether the lens should be moved toward or away from the camera, and the distance moved, in order to achieve focus and move the lens accordingly.

[0115] Phase detection relies on autofocus sensors that detect phase differences. In one example, light from an object is received by two or more small image sensors with multiple tiny lenses. When the light reaches both sensors, if the object is in focus, the light from the two extreme sides of the lens converges at the center of each sensor. Both sensors will have the same image, indicating that the object is indeed in focus. During the focusing process, the object is automatically focused by moving the lens.

[0116] Fig. 9 The figure is a focusing schematic diagram of an embodiment of a focusing method based on eye movement monitoring information disclosed in the present invention. Fig. 9 The numbers 1 to 4 in the figure represent the four states of the lens during the phase detection focusing process, wherein state 2 is the state in which the same beam of light is in focus. Fig. 9 The red and green areas represent the pixels on the two sensors used for focusing, and the black area represents the connection between the two pixels. Fig. 9 As shown in the figure, it can be seen that as the positions of the image sensor and the lens move relative to each other, a phase difference is generated between the curves of the two sensors. In the actual system, it is the lens rather than the sensor that moves during the focusing process. By detecting the position of the signal peak, the focusing system can know the direction of the offset, calculate the phase difference, derive the distance of the target, control the lens to move forward or backward, and automatically complete the focus.

[0117] The phase detection method has a fast focusing speed and can complete the focusing only once. Compared with the contrast detection method, the phase difference detection method focuses in one step and does not require repeated movement of the lens, but requires reserved masked pixels on the CCD (Charge-coupled Device) photosensitive element for phase detection.

[0118] In addition, the phase detection method also captures a small central area in the image lens. If the light in the central area can converge, the focus is completed. This method also causes the image outside the depth of field to be blurred, which makes the driver feel uncomfortable. The disclosed embodiment changes the existing focus on the central area to the focus on the injection area of ​​the human eye, so that the driver can see the image area of ​​interest clearly, thereby improving the image display effect and improving driving comfort and driving safety.

[0119] 3) Hybrid focus mode

[0120] Combining the contrast detection method and the phase detection method is called the hybrid focus mode. For example, to improve the focus accuracy, the phase detection method is used to quickly move the lens, and then the contrast detection method is used for fine-tuning. This mode gives full play to the advantages of both methods, and both the focus accuracy and the focus speed are significantly improved.

[0121] In one embodiment, Figure 3 In step S130, after controlling the image sensor to focus based on the gaze area so that the gaze area is clearly imaged, the method further includes:

[0122] Controls the depth of field range of the image shown in the monitor.

[0123] See also Fig.10 and Fig.11 The imaging principle of the lens is based on convex lens imaging. Through the combination of lenses, the light emitted or reflected by the object is imaged on the image plane. The image plane coincides with the silicon surface of the chip, that is, Fig.10 The plane where the sensor is horizontally located is also Fig.11 The focal plane in the image. The use of concave and convex lens combinations can effectively balance various aberrations such as spherical aberration, off-axis aberration, chromatic aberration, etc., and improve the image quality. The camera imaging process can be divided into the following steps: optical imaging, photoelectric conversion, signal processing, digital encoding and data transmission.

[0124] See also Fig.11 Depth of field refers to the acceptable clear area before and after the focus, that is, the clear range of the scene in the picture. In a photo with a small depth of field, only the focus part will be clearly displayed, and the areas outside the depth of field will appear very blurry; while in a photo with a large depth of field, all the scenes will appear very clear. The two have completely different effects in the picture.

[0125] The focus adjustment of the image sensor in step S130 will cause the depth of field to change. Therefore, in order to further improve the image display effect, the depth of field range of the displayed image can be further adjusted after this step. While adjusting the focus with eye movement, it is ensured that the image has a certain suitable depth of field range.

[0126] In one embodiment, controlling the depth of field range of an image displayed on a display screen includes:

[0127] By adjusting the image sensor's lens aperture, the depth of field range of the image is controlled.

[0128] See also Fig.12 and Fig.13One of the factors that affect the depth of field is the lens aperture. When the focal length and focus distance are the same, the larger the aperture, the smaller the depth of field; the smaller the aperture, the larger the depth of field. The aperture is a device used to control the amount of light that passes through the lens and enters the photosensitive surface inside the camera body. The aperture size is usually expressed in f-values. Fig.12 and Fig.13 The figure shows the image display effect of each aperture value. Fig.12 and Fig.13 As shown in the figure, the smaller the f-number of the aperture, the larger the aperture, the more obvious the background blur effect. Therefore, a small aperture can usually be used to obtain a large depth of field to show the depth of the scene, while also showing a clearer and wider sense of space.

[0129] In step S130, the image sensor is controlled to focus based on the gaze area so that the gaze area is clearly imaged. If the depth of field of the image becomes smaller, the aperture can be reduced to increase the depth of field to ensure that the range of the clear scene in the image is large enough.

[0130] In addition, under normal circumstances, the image captured by the image sensor will be affected by optics, and some areas of the entire image may be clear, while some areas are unclear. Multiple cameras can be set in the image sensor, and multiple cameras can be used to shoot the same position in the scene. The images taken by multiple cameras can be aligned and fused into one piece in combination with a multi-camera fusion algorithm, so that all areas or most areas of the entire image are clear. In the embodiment of the present disclosure, after focusing on the gaze area of ​​the human eye to make the gaze area clearly imaged, the multi-camera fusion algorithm can also be combined to improve the image clarity of other areas, thereby ensuring the improvement of the accuracy, security and stability of the comprehensive visual information.

[0131] like Fig.14 As shown, the present disclosure also provides a corresponding embodiment of a focusing device based on eye movement monitoring information. For the beneficial effects or technical problems solved by the device, please refer to the description of the method corresponding to each device, or refer to the description in the invention content, which will not be repeated here.

[0132] In an embodiment of the focusing device based on eye movement monitoring information, the device includes:

[0133] The receiving unit 100 is used to: receive the eye movement information of the driver acquired by the eye movement detection device;

[0134] The processing unit 200 is used to: determine the driver's gaze area on the display screen according to the eye movement information;

[0135] The adjustment unit 300 is used to control the image sensor to focus based on the gaze area so that the gaze area is clearly imaged.

[0136] In one implementation, the receiving unit 100 is configured to:

[0137] Collecting light signal information of a local spatial area and converting it into a pulse signal according to the light signal information; the light signal information includes optical flow information reflecting eye movement;

[0138] In response to the pulse signal, the pulse signal is input into an eye tracking model, and the eye tracking model outputs eye movement information corresponding to the pulse signal; the eye tracking model is generated based on a pulse neural network.

[0139] In one embodiment, the eye movement detection device and the display screen are arranged in the electronic vehicle rearview mirror; the image sensor is deployed at the rear, side or bottom of the vehicle.

[0140] In one implementation, the eye movement information includes image information of the driver's eyes and corneal reflection points.

[0141] In one embodiment, the processing unit 200 is configured to:

[0142] The image information of the driver's eyes and corneal reflection points is input into a pre-trained machine learning model, and the machine learning model outputs the driver's gaze area on the display screen.

[0143] In one implementation, the adjustment unit 300 controls the image sensor to focus, including:

[0144] Focusing is done by adjusting the lens position of the image sensor.

[0145] In one implementation, the adjustment unit 300 adjusts the lens position of the image sensor to perform focusing in a manner that includes at least one of a contrast detection method, a phase difference detection method, and a hybrid focus mode.

[0146] like Fig.15 As shown, in one embodiment, the above device further includes a control unit 400, and the control unit 400 is used to:

[0147] The image sensor is controlled to focus based on the gaze area, so that after the gaze area is clearly imaged, the depth of field range of the image displayed on the display screen is controlled.

[0148] In one embodiment, the control unit 400 is used to:

[0149] By adjusting the image sensor's lens aperture, the depth of field range of the image is controlled.

[0150] The focusing device based on eye movement monitoring information in the embodiment of the present disclosure corresponds to the above-mentioned focusing method embodiments based on eye movement monitoring information in the present disclosure in terms of specific implementation and beneficial technical effects. The relevant contents can be referenced to each other and will not be repeated here.

[0151] The present disclosure also provides a corresponding embodiment of an electronic vehicle-mounted rearview mirror. The electronic vehicle-mounted rearview mirror includes an eye movement detection device, a display screen, a processor, and multiple image sensors deployed at the rear, side, and bottom of the vehicle, wherein:

[0152] The eye movement detection device is arranged at a position adapted to the display screen, and is used to obtain the driver's eye movement information and send the eye movement information to the processor;

[0153] The processor is connected to the eye movement detection device to receive the eye movement information; and according to the eye movement information, determines the driver's gaze area on the display screen of the electronic vehicle rearview mirror;

[0154] The processor is also connected to the image sensor and is used to control the image sensor to focus based on the gaze area so that the gaze area is clearly imaged.

[0155] The electronic vehicle rearview mirror in the embodiment of the present disclosure and the above-mentioned focusing method embodiments based on eye movement monitoring information in the present disclosure correspond to each other in terms of specific implementation and beneficial technical effects. The relevant contents can be referenced to each other and will not be repeated here.

[0156] The present disclosure also provides a corresponding embodiment of a vehicle, wherein the vehicle comprises the electronic vehicle-mounted rearview mirror as described above, and the electronic vehicle-mounted rearview mirror serves as at least one of a left exterior rearview mirror, a right exterior rearview mirror, and an interior center rearview mirror.

[0157] Electronic rearview mirrors are important tools for drivers to obtain external information such as the rear, side, and bottom of the car during driving. Each car can usually be equipped with three rearview mirrors, namely:

[0158] 1) Left exterior rearview mirror: located on the left side of the cab, helps the driver observe the traffic conditions on the left side of the vehicle.

[0159] 2) Right exterior rearview mirror: located on the right side of the cab, helps the driver observe the traffic conditions on the right side of the vehicle.

[0160] 3) Center rearview mirror in the vehicle: located in front of the driver, facing the center of the road ahead, helping the driver observe the traffic conditions behind the vehicle, including the position and distance of the rear vehicles.

[0161] These three rearview mirrors work together to provide the driver with an all-round view, ensuring that the surrounding traffic conditions can be judged in a timely and accurate manner during driving, thereby ensuring driving safety.

[0162] Below, reference Fig.16The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0163] Fig.16 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0164] like Fig.16 As shown, the electronic device includes one or more processors and memory.

[0165] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0166] The memory may store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium, and the processor may run the computer program product to implement the focusing method based on eye movement monitoring information and / or other desired functions of the various embodiments of the present disclosure described above.

[0167] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0168] In addition, the input device may also include, for example, a keyboard, a mouse, and the like.

[0169] The output device can output various information to the outside, including determined distance information, direction information, etc. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0170] Of course, to simplify, Fig.16 Only some of the components related to the present disclosure in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device may further include any other appropriate components.

[0171] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the focusing method based on eye movement monitoring information according to various embodiments of the present disclosure described in the above part of this specification.

[0172] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0173] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the focusing method based on eye movement monitoring information according to various embodiments of the present disclosure described in the above part of this specification.

[0174] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0175] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0176] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0177] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0178] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0179] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0180] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0181] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A focusing method based on eye movement monitoring information, characterized in that: include: Receiving the driver's eye movement information acquired by the eye movement detection device; Determining a gaze area of ​​the driver on a display screen according to the eye movement information; Based on the gaze area, the image sensor is controlled to focus so that the gaze area is clearly imaged.

2. The method according to claim 1, characterized in that The receiving of the driver's eye movement information acquired by the eye movement detection device includes: Collecting light signal information of a local spatial area, and converting the light signal information into a pulse signal according to the light signal information; the light signal information includes optical flow information reflecting eye movement; In response to the pulse signal, the pulse signal is input into an eye tracking model, and the eye tracking model outputs eye movement information corresponding to the pulse signal; the eye tracking model is generated based on a pulse neural network.

3. The method according to claim 1, characterized in that The eye movement detection device and the display screen are arranged in an electronic vehicle-mounted rearview mirror; the image sensor is deployed at the rear, side or bottom of the vehicle.

4. The method according to claim 1, characterized in that The eye movement information includes image information of the driver's eyes and corneal reflection points.

5. The method according to claim 4, characterized in that The step of determining the driver's gaze area on the display screen according to the eye movement information includes: Image information of the driver's eyes and corneal reflection points is input into a pre-trained machine learning model, and the machine learning model outputs the driver's gaze area on the display screen.

6. The method according to claim 1, characterized in that The controlling the image sensor to focus comprises: Focusing is performed by adjusting the lens position of the image sensor.

7. The method according to claim 6, characterized in that The method of adjusting the lens position of the image sensor to focus includes at least one of a contrast detection method, a phase difference detection method and a hybrid focus mode.

8. The method according to any one of claims 1 to 7, characterized in that After controlling the image sensor to focus based on the gaze area so that the gaze area is clearly imaged, the method further includes: Controlling the depth of field range of an image displayed on the display screen.

9. The method according to claim 8, characterized in that The controlling the depth of field range of the image displayed on the display screen comprises: The depth of field range of the image is controlled by adjusting the lens aperture of the image sensor.

10. A focusing device based on eye movement monitoring information, characterized in that: include: A receiving unit, used to: receive the driver's eye movement information acquired by the eye movement detection device; A processing unit, configured to: determine a gaze area of ​​the driver on the display screen according to the eye movement information; The adjustment unit is used to control the image sensor to focus based on the gaze area so that the gaze area is clearly imaged.

11. An electronic vehicle rearview mirror, characterized in that: The electronic vehicle rearview mirror includes an eye movement detection device, a display screen, a processor, and a plurality of image sensors deployed at the rear, side, and bottom of the vehicle, wherein: The eye movement detection device is arranged at a position adapted to the display screen, and is used to obtain the eye movement information of the driver and send the eye movement information to the processor; The processor is connected to the eye movement detection device and is used to receive the eye movement information; and determine the driver's gaze area on the display screen of the electronic vehicle rearview mirror according to the eye movement information; The processor is also connected to the image sensor, and is used to control the image sensor to focus based on the gaze area, so that the gaze area is clearly imaged.

12. A vehicle, characterized in that: The vehicle is equipped with at least one electronic vehicle-mounted rearview mirror as claimed in claim 11, and the electronic vehicle-mounted rearview mirror serves as at least one of a left side exterior rearview mirror, a right side exterior rearview mirror and an interior center rearview mirror.

13. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor, further comprising the focusing device according to claim 10; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to control the focusing device to implement the method described in any one of claims 1-9.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the instructions are executed, the computer executes any one of the methods of claims 1-9.

15. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 9.

Images