Intelligent glasses and image processing method thereof

Abstract

The invention provides intelligent glasses and an image processing method thereof.The intelligent glasses comprise an eye movement tracking device, a first camera and a processor, and when the processor detects that the eye movement type is non-jumping, a target area image corresponding to a fixation point is determined according to eye movement tracking; and downsampling the first image acquired by the first camera to obtain a second image, and sending the second image and the target area image to the mobile terminal. AI identification and intelligent suggestion on an object watched by a user are realized through interaction between the intelligent glasses and the mobile terminal and in combination with eye movement tracking, the intelligent requirements of the intelligent glasses are enriched, and the communication data volume between the intelligent glasses and the mobile terminal is reduced through a mode of combining a high-definition local image with a low-definition panoramic image, so that the user experience is improved. The transmission rate between the intelligent glasses and the mobile terminal is improved, the waiting time of a user is shortened, meanwhile, the context background information of an identified object is fully combined, the AI task precision is improved, the requirements of more tasks are met, and the transmission efficiency and the task effect are both considered.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the technical field of wearable devices, and in particular, to smart glasses and an image processing method thereof. Background Art

[0002] As users' demand for the intelligence of wearable devices increases, smart glasses are no longer limited to conventional functions such as physiological parameter detection, music playback, calls, and recording. Some smart glasses can realize intelligent image recognition detection by integrating high-definition cameras, such as AI (Artificial Intelligence) object recognition, AI assistant and other functions.

[0003] In related technologies, the computing power of smart glasses is limited, resulting in a long image processing time. Users are clearly aware of the waiting process and the functional experience is poor. Summary of the invention

[0004] In view of this, the embodiments of this specification provide a pair of smart glasses and an image processing method, device, storage medium, and computer program product thereof.

[0005] In a first aspect, some embodiments of this specification provide a smart glasses, including:

[0006] An eye tracking device for collecting the gaze point of the user's eyes;

[0007] A first camera is used to capture a first image, where the first image represents an image of a scene within a user's field of view;

[0008] The processor is configured to, in response to detecting that the user's eye movement type is non-jumping, obtain a target area image from the segmentation of the first image based on the gaze point, downsample the first image to obtain a second image, send the target area image and the second image to a mobile terminal, and receive the image processing result sent by the mobile terminal.

[0009] In some embodiments, the eye tracking device includes a light source assembly and an infrared camera, wherein the light source assembly is used to emit an infrared light source toward the user's eyes, and the infrared camera is used to collect a light spot image of the user's eyes, wherein the light spot image includes a light spot formed on the user's eyes by the infrared light source emitted by the light source assembly;

[0010] The processor is configured to perform image detection on the light spot image, determine the relative position between the pupil center of the user's eye and the light spot, and determine the gaze point of the user's eye based on the relative position and a pre-calibrated light spot position.

[0011] In some embodiments, the infrared camera is further used to collect eye movement parameters of the user's eyes, where the eye movement parameters represent the eye movement speed of the user's eyes;

[0012] The processor is configured to receive the eye movement parameters collected by the infrared camera, and determine the eye movement type of the user's eyes based on the eye movement parameters.

[0013] In some embodiments, the smart glasses include a body, the body includes a lens frame and temples connected to both sides of the lens frame, the lens frame includes an inner wall facing the user's eyes and an outer wall away from the user's eyes;

[0014] The eye tracking device is arranged on the inner wall of the lens frame, and the first camera is arranged on the outer wall of the lens frame.

[0015] In some embodiments, the smart glasses further include a speaker, the processor is configured to generate a voice command based on the image processing result, and the speaker is used to play the voice command;

[0016] In some embodiments, the smart glasses further include a display device, and the display device is used to display the image processing result.

[0017] In a second aspect, an embodiment of the present specification provides an image processing method, which is applied to smart glasses, and the method includes:

[0018] In response to detecting that the eye movement type of the user is non-jumping, acquiring a first image within the user's visual field and a gaze point of the user's eyes;

[0019] Determine a target area on the first image based on the gaze point, and segment the target area to obtain a target area image;

[0020] Downsampling the first image to obtain a second image;

[0021] The target area image and the second image are sent to a mobile terminal bound to the smart glasses, and an image processing result sent by the mobile terminal is received.

[0022] In some implementations, the eye movement type includes beating and non-beating, and the process of detecting the user's eye movement type includes:

[0023] Collecting eye movement parameters of the user's eyes through an eye tracking device, wherein the eye movement parameters represent the eye movement speed of the user's eyes;

[0024] When the eye movement parameter is greater than or equal to a preset threshold, determining the eye movement type is a jump;

[0025] When the eye movement parameter is less than a preset threshold, the eye movement type is determined to be non-jumping.

[0026] In some implementations, the process of obtaining the gaze point of the user's eyes includes:

[0027] Using a light source assembly of an eye tracking device to emit an infrared light source to the user's eyes, and acquiring a light spot image of the user's eyes captured by an infrared camera of the eye tracking device, wherein the light spot image includes a light spot formed on the user's eyes by the infrared light source emitted by the light source assembly;

[0028] Performing image detection on the light spot image to determine the relative position of the pupil center of the user's eye and the light spot;

[0029] The gaze point of the user's eyes is determined based on the relative position and the pre-calibrated light spot position.

[0030] In some implementations, determining a target area on the first image based on the gaze point, and segmenting the target area to obtain a target area image, includes:

[0031] Determine the image position of the gaze point on the first image based on a pre-calibrated mapping relationship, wherein the mapping relationship represents a mapping relationship between a coordinate system where the gaze point is located and an image coordinate system of the first image;

[0032] Performing image detection on the first image, and determining an image area including the image position where the gaze point is located as the target area;

[0033] Performing image segmentation on the target area to obtain the target area image.

[0034] In some implementations, after receiving the image processing result sent by the mobile terminal, the method further includes:

[0035] generating a voice command based on the image processing result, and playing the voice command;

[0036] and / or,

[0037] The image processing result is displayed on a display device of the smart glasses.

[0038] In a third aspect, an embodiment of the present specification provides an image processing device, which is applied to smart glasses, and the device includes:

[0039] An image acquisition module, configured to acquire a first image within the user's visual field and a gaze point of the user's eyes in response to detecting that the user's eye movement type is non-jumping;

[0040] an image segmentation module, configured to determine a target area on the first image based on the gaze point, and segment the target area to obtain a target area image;

[0041] A downsampling module, configured to perform downsampling processing on the first image to obtain a second image;

[0042] The data communication module is configured to send the target area image and the second image to a mobile terminal bound to the smart glasses, and receive an image processing result sent by the mobile terminal.

[0043] In some embodiments, the eye movement type includes beating and non-beating, and the image acquisition module is configured to:

[0044] Collecting eye movement parameters of the user's eyes through an eye tracking device, wherein the eye movement parameters represent the eye movement speed of the user's eyes;

[0045] When the eye movement parameter is greater than or equal to a preset threshold, determining the eye movement type is a jump;

[0046] When the eye movement parameter is less than a preset threshold, the eye movement type is determined to be non-jumping.

[0047] In some embodiments, the image acquisition module is configured to:

[0048] Using a light source assembly of an eye tracking device to emit an infrared light source to the user's eyes, and acquiring a light spot image of the user's eyes captured by an infrared camera of the eye tracking device, wherein the light spot image includes a light spot formed on the user's eyes by the infrared light source emitted by the light source assembly;

[0049] Performing image detection on the light spot image to determine the relative position of the pupil center of the user's eye and the light spot;

[0050] The gaze point of the user's eyes is determined based on the relative position and the pre-calibrated light spot position.

[0051] In some embodiments, the image segmentation module is configured to:

[0052] Determine the image position of the gaze point on the first image based on a pre-calibrated mapping relationship, wherein the mapping relationship represents a mapping relationship between a coordinate system where the gaze point is located and an image coordinate system of the first image;

[0053] Performing image detection on the first image, and determining an image area including the image position where the gaze point is located as the target area;

[0054] Performing image segmentation on the target area to obtain the target area image.

[0055] In a fourth aspect, an embodiment of this specification provides a storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in any of the above embodiments.

[0056] In a fifth aspect, an embodiment of this specification provides a computer program product, which, when executed by a computer, implements the method described in any of the above embodiments.

[0057] The smart glasses of the embodiment of the present specification include an eye tracking device, a first camera and a processor. When the processor detects that the eye movement type is non-jumping, it determines the target area image corresponding to the gaze point according to the eye tracking, and downsamples the first image captured by the first camera to obtain a second image, and sends the second image and the target area image to the mobile terminal. Through the interaction between the smart glasses and the mobile terminal, combined with eye tracking, AI recognition and intelligent suggestions for the object the user is looking at are realized, enriching the intelligent needs of the smart glasses, and by combining high-definition local images with low-definition panoramic images, the amount of communication data between the mobile terminal is reduced, the transmission rate between the smart glasses and the mobile terminal is increased, and the user's waiting time is reduced. At the same time, the contextual background information of the identified object is fully combined to improve the accuracy of AI tasks, meet the needs of more tasks, and take into account both transmission efficiency and task effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a schematic diagram of smart glasses according to an exemplary embodiment of the present specification.

[0059] Figure 2 It is a schematic diagram of the interaction among smart glasses, a mobile terminal, and a cloud server in an exemplary implementation of this specification.

[0060] Figure 3 It is a block diagram of the structure of smart glasses according to an exemplary embodiment of this specification.

[0061] Figure 4 is a flowchart of an image processing method according to an exemplary embodiment of the present specification.

[0062] Figure 5 is a flowchart of an image processing method according to an exemplary embodiment of the present specification.

[0063] Figure 6 is a flowchart of an image processing method according to an exemplary embodiment of the present specification.

[0064] Figure 7 is a schematic diagram of an image processing method according to an exemplary embodiment of the present specification.

[0065] Figure 8is a schematic diagram of an image processing method according to an exemplary embodiment of the present specification.

[0066] Fig. 9 is a flowchart of an image processing method according to an exemplary embodiment of the present specification.

[0067] Fig.10 is a schematic diagram of an image processing method according to an exemplary embodiment of the present specification.

[0068] Fig.11 is a block diagram of the structure of an image processing device according to an exemplary embodiment of the present specification. DETAILED DESCRIPTION

[0069] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0070] With the development of smart wearable devices, smart glasses products are emerging in an endless stream. Currently, smart glasses products can be divided into two categories. One category of products is mainly for AR (Augmented Reality) / VR (Virtual Reality) and other head-mounted entertainment scenarios. Such products are generally large in size and have high computing power performance, suitable for short-term gaming and entertainment. The other category of products mainly realizes functions such as health detection, music playback, and making and receiving calls. Such products need to meet the daily wearing needs of users, and require the appearance, size and weight to be the same as ordinary glasses, and the computing power performance of the device itself is limited.

[0071] Nowadays, users have higher and higher demands for the intelligence of smart glasses. They are no longer satisfied with conventional functions such as physiological parameter detection (such as heart rate, blood oxygen saturation, blood pressure, etc.), music playback, calls, recording, etc. Therefore, some smart glasses integrate AI (Artificial Intelligence) functions to realize functions such as AI object recognition and AI assistant.

[0072] In life scenarios, users often have the need to identify objects, such as the names of flowers and plants, pet species, the usage of medicines, the calories of food, etc. At present, some of these needs can be met on mobile terminals. For example, taking a smartphone as an example, the user can open the mobile phone camera App (Application), turn on the AI ​​object recognition function, and aim the mobile phone viewfinder at the object to be identified, so that the object information, such as the name of flowers and plants, pet species, etc., can be displayed in the viewfinder. Or the user can take an image with the camera and use AI object recognition to identify the object information in the captured image.

[0073] In the above mobile terminal scenario, users need to actively use mobile terminals to collect images of objects to be identified, and the operation process is complicated and cumbersome. If the above functions can be transplanted into smart glasses, users only need to look at the object normally, and smart glasses can recognize the object information and inform the user through voice or display, which will greatly simplify user operations and free the user's hands.

[0074] For example Figure 1 In the example, an external camera can be set on the smart glasses. The function of the external camera is to collect environmental images within the user's field of view. For example, when the user sees a flower and wants to know the name of the flower and related information, the smart glasses can collect scene images through the external camera, and then use the image processing algorithm in combination with the neural network model to obtain relevant information about the flower, and then feedback the results to the user through voice or text display. For another example, some users have a need to control sugar. When users buy fruits, they hope to know the sugar content of each fruit, so as to guide themselves to buy suitable fruits. At this time, the smart glasses can collect fruit images through the external camera, and then use the image processing algorithm in combination with the neural network model to obtain the sugar content information of one or more fruits.

[0075] In the above example scenario, compared with traditional mobile terminals, smart glasses greatly facilitate user operation. Since smart glasses are worn directly on the user's head and follow the movement of the user's head, images within the user's field of view can always be captured. The user does not need to manually adjust the angle and take pictures, freeing the user's hands and improving operational convenience.

[0076] However, in order to realize AI functions, it is inevitable to use corresponding AI models. AI models refer to neural network models used in the field of artificial intelligence. AI models are trained with a large amount of data and perform well in tasks such as NLP (Natural Language Processing) and CV (Computer Vision).

[0077] AI models have large parameters and volumes. For example, large generative models can have parameters in the billions or even trillions. Smart glasses and mobile terminals have limited performance, so models cannot be deployed locally. Generally, AI models are deployed on cloud servers. Figure 2 As shown, the smart glasses can establish a binding relationship with the mobile terminal through near field communication, and the near field communication method may include Bluetooth, WiFi, etc. The mobile terminal establishes a communication connection with the cloud server through a wide area network, such as a cellular network. When implementing the aforementioned AI function, the smart glasses are required to upload the collected images to the mobile terminal, and then the mobile terminal sends them to the cloud server. After the cloud server processes the image, it sends the result to the mobile terminal, and the mobile terminal sends it to the smart glasses.

[0078] For image processing tasks, image clarity is the basis for ensuring task accuracy. Moreover, as users pursue high-definition images, the external cameras of smart glasses are generally high-definition cameras, and the collected scene images are high-definition images with large image volumes. As a result, when the smart glasses transmit images to mobile terminals, the transmission process is slow, causing the entire business process to be slowed down. Users can clearly perceive the waiting time, and the actual experience is not good.

[0079] In addition, in actual scenarios, the captured images will not only include objects that the user is interested in, but may also include other objects. For example, in one example, a user stands in front of a fruit stall, and the user only wants to know the sugar content of the fruit "watermelon", while the images captured by the smart glasses may also include other fruits. After sending the image to the cloud server, the AI ​​model needs to extract and process the features of all the fruits in the image, which undoubtedly increases the computing burden and slows down the business process. If the user's interest is segmented from the image before entering the model, and only the area of ​​interest is sent to the model, the contextual information of the object to be identified is lost due to the lack of background images, and the output results are not accurate in some task scenarios.

[0080] It can be seen that in the related art, the image transmission efficiency between smart glasses and mobile terminals is low, and the image processing of the user's area of ​​interest is poor, resulting in poor AI service effects of smart glasses. Based on this, the embodiments of this specification provide smart glasses, as well as image processing methods, devices, storage media and computer program products for the smart glasses.

[0081] In some embodiments, the present specification provides a pair of smart glasses, which include a body and various electrical components disposed on the body.

[0082] See for example Figure 1As shown, the smart glasses body includes a lens frame and temples connected to both sides of the lens frame. The lens frame refers to a bearing frame for assembling transparent lenses. The lenses can be selected by users according to their own needs, so that they are suitable for daily wear by different users. This specification does not limit this. The temples can be connected to the lens frame through hinges, so that the temples can be folded or unfolded. In the implementation mode of this specification, the electrical components can be arranged inside the lens frame and the temples.

[0083] Figure 3 The structure diagram of the smart glasses in some embodiments of this specification is shown below. Figure 3 The electrical components and structure of smart glasses are explained.

[0084] See also Figure 3 As shown, the smart glasses include a processor, a first camera, an eye tracking device, a communication module, an ultraviolet sensor, a display device, a speaker, a microphone, an IMU sensor, a heart rate / body temperature / EEG sensor, touch buttons, etc.

[0085] The processor may include one or more processing units, for example: the processor may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a video processing unit (video processing unit, VPU) controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural network processor (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. In some embodiments, the processor can also be a microcontroller unit (Microcontroller Unit, MCU).

[0086] A memory may also be provided in the processor for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. The memory may store instructions or data that the processor has just used or is cyclically used. If the processor needs to use the instruction or data again, it may be directly called from the memory. Repeated access is avoided, the waiting time of the processor is reduced, and the efficiency of the system is improved.

[0087] In some embodiments, the processor can be disposed on a main board disposed inside the temples of the smart glasses.

[0088] The first camera refers to a device provided on the smart glasses for realizing image acquisition. The first camera may include an optical lens module, which is composed of a plurality of lenses combined along an optical axis. It may also include a photosensitive element, such as a CMOS (Complementary Metal Oxide Semiconductor) sensor, and an ISP (Image Signal Processing) chip, etc.

[0089] In the implementation mode of this specification, the first camera can collect scene images within the user's field of view. For example, in one example, see Figure 1 As shown, the first camera can be the external camera of the smart glasses, that is, the first camera is arranged on the outer wall of the lens frame facing the outside, so that the first camera can collect the external environment image. It can be understood that after the user wears the smart glasses, the user's field of view is through the lens toward the outside, so the environment image collected by the first camera is the scene image within the user's field of view.

[0090] The eye tracking device is used to track the movement of the user's eyes, and then determine the gaze point of the human eye. For example, the pupil position can be located through image processing technology, the coordinates of the pupil center can be obtained, and then the gaze point of the human eye can be calculated. In the image processing method below in this specification, the algorithm process of determining the gaze point by eye tracking is described.

[0091] In the disclosed embodiment, the eye tracking device can be arranged on the inner side wall of the lens frame, and the eye tracking device is facing the user's eyes, so that the image of the user's eyes can be collected to realize eye tracking. It is worth noting that the eye tracking device can be arranged only near one of the eyes, or the eye tracking device can be arranged near both eyes respectively, and this specification does not limit this.

[0092] In some embodiments, the eye tracking device includes a light source assembly and an infrared camera. The light source assembly includes one or more infrared emission sources. The function of the infrared emission source is to emit infrared light toward the human eye. Since infrared light is invisible to the human eye, the infrared light source will not affect the human eye, and eye tracking without user perception can be achieved. The infrared emission source emits an infrared light source toward the human eye. After the infrared light source irradiates the human eye, it will be reflected in the cornea or iris of the human eye. The infrared camera can collect the reflected light and the image of the human eye to obtain a light spot image including the human eye. The light spot image includes the image of the reflected light (that is, the light spot) and the image of the human eye. According to the light spot image, the direction of the optical axis of the pupil can be determined, thereby obtaining the direction of the user's line of sight and determining the gaze point.

[0093] In some embodiments, the infrared camera may also collect eye movement parameters of the user's eyes, which represent the eye movement speed of the user's eyes. For example, in one example, the eye movement parameters may include eye movement speed parameters and acceleration parameters of the user's eyeballs. Through the eye movement parameters, the processor may determine the user's eye movement type, that is, whether the user is currently in a gaze state. This is described in the following embodiments of this specification.

[0094] The UV sensor is a device that converts UV signals into electrical signals. In some embodiments of the present specification, the UV sensor can be disposed on the outer wall of the lens frame, so that the UV sensor can detect the UV intensity of the external environment. The processor can remind the user to pay attention to sun protection or notify the user of the UV level based on the UV intensity detected by the UV sensor, so as to avoid the user being exposed to a strong UV environment for a long time.

[0095] The communication module refers to a related circuit module for data exchange between smart glasses and external electronic devices. In the implementation manner of this specification, the communication module can be a wired communication module or a wireless communication module. Taking the wired communication module as an example, the wired communication module may include a data interface. The data interface may be, for example, a USB (Universal Serial Bus) interface, a Micro USB interface, a Type-C interface, etc., and a wired communication connection with other electronic devices can be achieved through the plug-in connection between the data cable and the data interface. Taking the wireless communication module as an example, the wireless communication module may include, for example, a Bluetooth module, a WiFi module, and a corresponding antenna structure, etc., and a wireless communication connection can be established with other electronic devices through the wireless communication module, for example Figure 2 In the example, the smart glasses can establish a wireless communication connection with the mobile terminal through the wireless communication module.

[0096] In some embodiments, the communication module may be disposed on a main board, and the main board is disposed inside the temple.

[0097] The display device is used to present images to the user under the control of the processor. In some embodiments, the display device may be a projection device disposed on the temple, so that the projection device can project the image signal sent by the processor onto the lens, and the user can see the displayed image on the lens. In other embodiments, the display device may be a transparent display screen as a lens, so that the image signal is directly displayed on the lens, and the user can see the displayed image on the lens. It is understandable that there are many types of display devices for displaying images on lenses, and they are not limited to the above examples.

[0098] Heart rate / body temperature / EEG sensor refers to an electronic device used to detect the physiological parameters of the user. For example, the heart rate sensor can be set on the temple, and the detection end is set on the inner wall of the temple, so that after the user wears the smart glasses, the skin behind the ear can be close to the detection end of the heart rate sensor to achieve heart rate detection. Body temperature and EEG detection can be similar.

[0099] IMU (Inertial Measurement Unit) sensor refers to a device used to detect the three-axis posture and acceleration of the device. The IMU sensor may include an accelerometer and a gyroscope. When the user's head posture changes, the smart glasses follow the head movement, and the IMU sensor can detect the posture and acceleration parameters of the smart glasses. The processor can calculate the posture of the device in real time based on the posture and acceleration parameters.

[0100] In some embodiments, when a user is wearing smart glasses and is in a stationary state, posture changes detected by an IMU sensor can reflect the user's heart rate and breathing state, thereby realizing the detection of physiological parameters such as the user's heart rate and breathing.

[0101] Speakers and microphones are used to realize the audio functions of smart glasses, such as music playback, recording, etc. Speakers are also called "horns" and are used to convert audio electrical signals into sound signals. Smart glasses can listen to music or answer calls through speakers. Microphones are also called "microphones" and "microphones". They are used to convert sound signals into electrical signals. Smart glasses can use microphones to realize sound collection, voice conversations, etc.

[0102] In the implementation manner of this specification, multiple speakers and multiple microphones can be set on the smart glasses, and the multiple speakers or multiple microphones can achieve the same function or different functions. For example, in one example, a speaker can be set at the position behind the ear of each temple, and the two speakers form a stereo speaker to provide the user with a higher auditory experience. The microphone can also include a microphone array set at multiple positions of the smart glasses, and the user's voice and environmental noise are collected by the microphone array respectively, so as to achieve call noise reduction.

[0103] Touch buttons refer to the interactive buttons between users and smart glasses. Touch buttons can be various forms of buttons, such as physical buttons, pressure-sensitive buttons, etc. Touch buttons can realize various functions, such as volume adjustment, controlling music playback and pause, switching to the previous / next song, answering / hanging up calls, turning on / off the eye tracking function, etc. In some embodiments, the touch buttons can be set at any position of the smart glasses as long as the user can operate the touch buttons. For example, in one example, the touch buttons are set on the temples.

[0104] In some embodiments, the smart glasses may further include a second camera (not shown in the drawings). The second camera may be disposed on the inner wall of the lens frame, or at the connection position between the lens frame and the temple. The second camera faces the human eye. The function of the second camera is to capture eye images, and the user's emotions may be recognized based on the eye images.

[0105] The above describes some of the electrical components included in the smart glasses. It is understandable that the electrical components of the smart glasses are not limited to the above examples, and may include more or fewer devices. For example, on the basis of the above, the smart glasses may also include batteries, LEDs (Light-Emitting Diodes), etc., which will not be elaborated in the present disclosure.

[0106] In some embodiments, the image processing method provided in this specification can be applied to the smart glasses in any of the aforementioned embodiments and executed by the processor of the smart glasses. Figure 4 Describe the method process.

[0107] like Figure 4 As shown, in some embodiments, the image processing method exemplified in this specification includes:

[0108] S410: In response to detecting that the eye movement type of the user is non-jumping, acquiring a first image within the user's field of view and a gaze point of the user's eyes.

[0109] Eye movement type refers to the type of human eye activity. In the implementation of this specification, the speed of human eye movement is used to represent the eye movement type, and the eye movement type can be divided into beating and non-beating. Beating refers to the rapid movement of the eyeball, and the line of sight moves rapidly in a short period of time. For example, when a user is rapidly scanning the environment, the eyeball moves rapidly to observe the surrounding environment. At this time, the eyeball movement speed is very fast, and the corresponding eye movement type is beating. Non-beating is the opposite, indicating that the human eyeball movement speed is very slow or still. For example, when a user is looking at an object, the line of sight of the eyeball is fixed and the eyeball is in a stationary state; for example, when a user slowly moves the line of sight while reading a book, the eyeball moves very slowly at this time. This type of eye movement in which the line of sight is still or moving slowly is non-beating.

[0110] In some embodiments, non-jumping can be further divided into two types: fixation and gentle movement. Fixation means that the human eyeball is in a static state and the line of sight remains unchanged. Slow movement means that the human eyeball is in slow motion and the line of sight moves slowly. Of course, it can be understood that the eye movement type can also include only jerking and non-jumping.

[0111] In the implementation mode of this specification, the smart glasses can detect the eye movement type of the user in real time. If the eye movement type is jumping, it means that the user is not looking at an object at this time, and there is no need to identify the object, and there is no need to start the image processing method of this specification. On the contrary, if the eye movement type is detected as non-jumping, it means that the user is looking at an object at this time, and the user has a need to identify the object, so the method steps of the image processing method of this specification can be started.

[0112] In some embodiments, an eye tracking device may be used to identify the user's eye movement velocity, and based on the eye movement velocity, determine whether the eye movement type is non-jumping. Figure 5 Provide explanation.

[0113] like Figure 5 As shown, in some embodiments, the image processing method exemplified in this specification, the process of detecting the user's eye movement type includes:

[0114] S4101. Collect eye movement parameters of the user through an eye tracking device.

[0115] S4102: When the eye movement parameter is greater than or equal to a preset threshold, determine the eye movement type as a jump; when the eye movement parameter is less than the preset threshold, determine the eye movement type as a non-jump.

[0116] In the implementation manner of this specification, eye movement parameters refer to relevant parameters used to characterize the user's eye movement speed, such as eye movement speed parameters, eye movement acceleration parameters, etc. Taking eye movement speed parameters as an example, eye movement speed can be expressed by the displacement of the center of the pupil of the eye per unit time.

[0117] In combination with the above, it can be known that the eye tracking device includes an infrared camera, which can be used to capture the user's eye image, and identify the user's pupil center coordinates through image detection, and calculate the eye movement speed parameters based on the change of the pupil center coordinates per unit time.

[0118] In the implementation mode of this specification, a corresponding threshold or threshold range can be set in advance for the eye movement speed parameter, and the interval of the eye movement type can be defined by the threshold range. For example, in one example, the threshold range of the eye movement speed can be shown in the following table 1:

[0119] Table 1

[0120] Eye movement velocity parameters Eye movement type ≥Th1 beat <Th1 Non-jumping

[0121] In the example of Table 1, if the detected eye movement speed parameter is greater than or equal to the preset threshold value Th1, it means that the current eye movement speed is very fast, and the eye movement type of the eyeball can be determined as jumping. On the contrary, if the detected eye movement speed parameter is less than the preset threshold value Th1, it means that the current eye movement speed is slow, and the eye movement type of the eyeball can be determined as non-jumping.

[0122] In the example of Table 1 above, the eye movement types are only taken as "jump" and "non-jump". In combination with the above, it can be seen that in other implementations, the eye movement types can be further subdivided into "jump", "smooth movement" and "gaze". At the same time, the threshold ranges can be divided in advance for these three eye movement types. For example, in an example, the threshold range of eye movement speed can be shown in Table 2 below:

[0123] Table 2

[0124]

[0125]

[0126] In the example of Table 2, if the detected eye movement speed parameter is greater than or equal to the preset threshold Th1, it means that the current eye movement speed is very fast, and the eye movement type of the eyeball can be determined as jumping. If the detected eye movement speed parameter is greater than the preset threshold Th1 and less than or equal to the preset threshold Th2, it means that the eye movement speed of the eyeball is slow and in a slow movement state, and the eye movement type can be determined as gentle movement. If the detected eye movement speed parameter is less than the preset threshold Th2, it means that the current eye movement speed is very slow, and it can be considered that the user is in a gaze state, and the eye movement type of the eyeball can be determined as gaze.

[0127] Those skilled in the art can understand that, in the above examples, the method of this specification can be executed when the eye movement type is determined to be non-jumping according to the example in Table 1, or the method of this specification can be executed when the eye movement type is determined to be fixation according to the example in Table 2, without limitation.

[0128] When it is determined that the user's eye movement type is non-jumping, it means that the user needs to recognize an object in the actual scene. At this time, the first camera mentioned above can be called to capture the first image. Figure 1 As shown, the first camera refers to an external camera that is arranged on the smart glasses and faces the outside world, so that the first image captured by the first camera is the image within the user's field of view.

[0129] It is understandable that when a user wants to identify an object, he or she will generally look at the object, so that the object will appear in the user's field of view. At this time, the first image captured by the first camera is the image containing the object to be identified. However, in actual scenes, the first image captured by the first camera, in addition to the object to be identified, often also contains a lot of other background or environmental information. In order to accurately identify the object to be identified that the user is concerned about, the user's eye gaze point can be combined, and the object pointed to by the gaze point is the object to be identified.

[0130] Therefore, in the implementation mode of this specification, while collecting the first image, it is also necessary to use an eye tracking device to identify the gaze point of the user's eyes. Figure 6 The implementation method is described.

[0131] like Figure 6 As shown, in some embodiments, the image processing method exemplified in this specification, the process of determining the gaze point of the user's eyes includes:

[0132] S4111. Use the light source component of the eye tracking device to emit an infrared light source to the user's eyes, and obtain a light spot image of the user's eyes captured by the infrared camera of the eye tracking device.

[0133] S4112: Perform image detection on the light spot image to determine the relative position of the pupil center of the user's eye and the light spot.

[0134] S4113. Determine the gaze point of the user's eyes based on the relative position and the pre-calibrated light spot position.

[0135] For ease of understanding, the following Figure 7 and Figure 8 The working principle of the eye tracking device is explained.

[0136] Figure 7 shows the cross-sectional structure of the human eyeball, see Figure 7 As shown, the eyeball includes cornea 601, iris 602, pupil 603, sclera 604, lens 605, and retina 606. The cornea 601 has a high reflectivity to the light irradiated by the light source, so a clearer reflection point can be formed after the light source irradiates. The cornea 601 is the transparent part in the front of the eyeball and is the first barrier for light to enter the eyeball. The central 3mm of the outer surface of the cornea 601 is a spherical arc surface, called the optical zone, and the peripheral curvature radius gradually increases, presenting an aspherical shape. Figure 7 In the eyeball structure model shown, the cornea 601 is assumed to be a spherical arc surface.

[0137] Since the iris 602 covered behind the cornea 601 is a disc-shaped membrane, the area where it is located is dark and circular, and there is a hole in the center called the pupil 603. The pupil 603 is a small round hole in the center of the iris in the eyes of animals or humans, which is the channel for light to enter the eye. The center point of the pupil 603 is the viewpoint. The lens 605 is a biconvex transparent tissue located behind the iris 602. Its shape and function are similar to a convex lens, and it can clearly reflect the images of objects near and far on the retina 606. The retina 606 is the photosensitive part of the eyeball, and external objects are imaged on the retina 606. The sclera 604 (also known as the white of the eye) has a low reflectivity to light irradiated by the light source. The sclera 604 is one of the main components of the eyeball wall, located at the junction with the cornea, with a tough structure, and has the function of supporting and protecting the intraocular tissue.

[0138] When detecting the gaze point, the eye tracking device can use the light source component to emit an infrared light source to the human eye. After the infrared light source is irradiated to the human eye, it will be reflected in the cornea or iris and other areas of the human eye. The infrared camera can collect the user's eye image, which includes the human eye image and the image of the reflected light. The image of the reflected light forms a light spot on the human eye, thus becoming a light spot image.

[0139] For example, in one example, the spot image captured by the infrared camera can be Figure 8 As shown, the light spot 903 is the image of the infrared light source on the eyeball through the cornea or iris range, and the human eye contour 901, the iris range 902, and the pupil 904 are the human eye image. Of course, it can be understood that the light spot image will also include other tissues and parts of the user's eye, such as eyelids, eyelashes, etc., which will not be shown in this manual.

[0140] In the implementation manner of this specification, after acquiring the light spot image, the processor can perform image detection on the light spot image to determine the pupil center and the position information of each light spot. Figure 8 In the example, the center point coordinates of the pupil 904 and the coordinates of each light spot are determined by performing image detection on the light spot image. After the center point coordinates of the pupil 904 and the coordinates of the light spot are determined, the relative position between the pupil center and the light spot can be obtained.

[0141] When the eye tracking device is mounted on the smart glasses, the eye tracking device can be calibrated. The purpose of calibration is to establish a mapping relationship between the position of the light spot on the image and the position in the world coordinate system, that is, the pre-calibrated light spot position is the position of the light spot corresponding to the real world. Therefore, after the relative position of the pupil center and the light spot is detected, the position information of the pupil center in the world coordinate system can be determined based on the pre-calibrated light spot position, and this position information is the gaze point of the user's eyes.

[0142] S420: Determine a target area on the first image based on the gaze point, and segment the target area to obtain a target area image.

[0143] In the above S410, the first camera is used to capture a first image containing an object to be identified, and the eye tracking device is used to detect the gaze point of the user's eyes. Next, the area where the object to be identified is located can be determined from the first image according to the gaze point of the user's eyes. Fig. 9 Provide explanation.

[0144] like Fig. 9 As shown, in some embodiments, the image processing method exemplified in this specification, the process of determining and segmenting the target area image includes:

[0145] S421. Determine an image position of a gaze point on the first image based on a pre-calibrated mapping relationship.

[0146] S422: Perform image detection on the first image, and determine an image area including the image position where the gaze point is located as a target area.

[0147] S423, performing image segmentation on the target area to obtain a target area image.

[0148] In the implementation manner of this specification, the target area refers to a local area on the first image, which includes the object that the user is looking at, that is, the object to be identified. The purpose of image segmentation is to segment the image area of ​​the object to be identified from the first image.

[0149] First, it is necessary to determine the image coordinates corresponding to the gaze point on the first image. For example, the mapping relationship between the image coordinate system of the first image and the world coordinate system can be pre-calibrated. Then, the coordinates of the gaze point are spatially transformed using the mapping relationship. The gaze point coordinates can be converted into the image coordinate system to determine the image coordinates of the gaze point on the first image.

[0150] For example, in one example, the first image captured may be Fig.10 As shown, the first image includes two kinds of fruits: an apple and an orange. Through the above image detection process, the image position of the gaze point on the first image is determined as Fig.10 As shown in , it means that the user is looking at the orange.

[0151] After that, it is necessary to detect the objects on the image based on the image position of the gaze point, so as to determine the target area where the gaze point is located. Fig.10For example, the processor can perform edge feature extraction on the first image. There are many edge feature extraction algorithms, including but not limited to the Sobel algorithm, the Canny algorithm, etc., which are not limited in the present disclosure. The purpose of edge feature extraction is to determine the various objects included in the first image, such as Fig.10 In this example, after edge feature extraction, the image edges of objects such as apples, oranges, plates, tables, and backgrounds in the first image can be obtained, and the image content can be divided into multiple regions using the image edges. Finally, combined with the image position of the gaze point, the image region where the "orange" is located is determined as the target region.

[0152] After determining the target area, the target area can be segmented to obtain an image of the target area containing only "oranges".

[0153] S430: Downsample the first image to obtain a second image.

[0154] It is worth noting that in the implementation of this specification, after the target area image is segmented, it is not just the target area image that is sent to the mobile terminal. This is because, in the field of image processing, the environment and background information around the object also contain rich semantic features. In some image processing tasks, combining the contextual information of the background area can better assist the image detection task, which can improve the accuracy of the image processing task on the one hand, and realize more task types on the other hand.

[0155] For example, if in the aforementioned image segmentation process, the segmented target area image is a blank area with no practical meaning, if only the target area image is transmitted to the backend model, the model cannot accurately recognize the user's intention. In actual scenarios, the target area image may contain rich image content around it. If it can be combined with a panoramic image, the model can more accurately recognize objects and user intentions.

[0156] For example, in another example, Fig.10 Taking the first image shown as an example, in a business scenario, although the user is looking at oranges, the user may want to know the sugar content of each fruit. If only the orange area being looked at is segmented and sent to the back-end model as the target area image, the model can only get the sugar content result of the orange and feedback it to the user, but cannot make intelligent recommendations for other fruits.

[0157] Based on this, in the implementation of this specification, in addition to transmitting the target area image to the back-end model, the panoramic image also needs to be transmitted to the back-end model. However, combined with the above, it can be seen that the panoramic image has a high definition and a large amount of data, so the transmission speed between the smart glasses and the mobile terminal is slow, resulting in a long waiting time for the user.

[0158] Therefore, in the implementation manner of this specification, by combining high-definition local images with low-definition panoramic images, the image area of ​​interest to the user and the panoramic image can be sent to the back-end model. At the same time, by reducing the resolution of the panoramic image, the amount of transmitted data can be reduced, thereby improving AI business efficiency and reducing user waiting time.

[0159] Specifically, in some embodiments, after segmenting the target area image, the processor may downsample the first image to obtain the second image. Image downsampling refers to the process of reducing the image resolution. Downsampling methods may include, for example, pixel deletion, pixel resampling, etc. Taking pixel resampling as an example, an interpolation algorithm may be used to calculate n adjacent pixels as one pixel, thereby greatly reducing the amount of data. The image obtained after downsampling is the second image.

[0160] S440: Send the target area image and the second image to a mobile terminal bound to the smart glasses, and receive the image processing result sent by the mobile terminal.

[0161] According to the foregoing, in the implementation mode of this specification, the high-definition target area image and the low-definition second image after downsampling are sent to the mobile terminal together, combined with Figure 2 As shown, when the computing power of the mobile terminal is sufficient, the AI ​​model deployed on the mobile terminal can be used to perform image processing based on the target area image and the second image, obtain the corresponding image processing results and return them to the smart glasses. When the computing power of the mobile terminal is insufficient, the mobile terminal can forward the target area image and the second image to the cloud server, and the AI ​​model deployed on the cloud server can perform image processing based on the target area image and the second image, obtain the corresponding image processing results and return them to the smart glasses.

[0162] It can be understood that the target area image refers to the object that the user is looking at, which is the object that the user is concerned about in the first image. Therefore, the target area image of the object is a high-definition image to ensure the recognition accuracy of the object. At the same time, the second image is a panoramic image, which serves as background information to assist the image processing task. Therefore, by downsampling to reduce the resolution of the panoramic image, the amount of transmitted data can be reduced, the transmission rate between the smart glasses and the mobile terminal can be increased, and the user's waiting time can be reduced.

[0163] In other words, compared with the solution of directly sending the first image, the implementation of this specification can effectively reduce the amount of data, improve the transmission efficiency between the smart glasses and the mobile terminal, and improve the user experience. At the same time, compared with the solution of only sending the segmented target area image, the contextual background information of the identified object can be fully combined to improve the accuracy of the AI ​​task, and can meet the needs of AI tasks that require full image information. The implementation of this specification takes into account both transmission efficiency and task effect.

[0164] It is worth noting that the implementation methods of this specification do not limit the specific functional implementation. The solution of this specification can be applied to any business scenario based on image processing. Several business scenarios are used as examples below for explanation.

[0165] 1. Diet and health management

[0166] The smart glasses can capture a first image containing food such as dishes, fruits, and cakes, and then based on the aforementioned image processing method, segment the target area image of the food that the user is looking at, and downsample the original first image to obtain a second image, and then send the second image and the target area image of the food to the mobile terminal.

[0167] The mobile terminal forwards the second image and the target area image to the cloud server. The cloud server calls the AI ​​model and sends the second image and the target area image as input to the AI ​​model. The AI ​​model outputs the corresponding calories (such as calorie value), nutritional components (such as sugar content), dish names, popular cooking methods, etc. of the food through image detection as image processing results.

[0168] The cloud server sends the image processing results to the mobile terminal, and the mobile terminal forwards the image processing results to the smart glasses, so that the user can obtain information such as food calories, nutritional ingredients, preparation methods, and dish names on the smart glasses. The user can freely choose which specific information the image processing results include. The user can pre-set the required result information on the mobile terminal, so that only the image processing result information set by the user can be displayed.

[0169] 2. Medication guidance

[0170] The smart glasses can capture a first image containing the name of a medicine (such as a medicine box, a medicine packaging bag, etc.), and then based on the aforementioned image processing method, segment the target area image of the medicine that the user is looking at, and downsample the original first image to obtain a second image, and then send the second image and the target area image of the food to the mobile terminal.

[0171] The mobile terminal forwards the second image and the target area image to the cloud server. The cloud server calls the AI ​​model and sends the second image and the target area image as input to the AI ​​model. The AI ​​model outputs the corresponding instructions for use, medication guidance, precautions, etc. of the drug through image detection, or can further provide health advice.

[0172] The cloud server sends this information as image processing results to the mobile terminal, and the mobile terminal forwards the image processing results to the smart glasses, so that users can get instructions for use, medication instructions, precautions, health advice and other information on the smart glasses. The user can freely choose which specific information the image processing results include. The user can pre-set the required result information on the mobile terminal, so that only the image processing result information set by the user can be displayed.

[0173] 3. Emotional counseling

[0174] Smart glasses can use the infrared camera of the eye tracking device to collect user eye images, and directly send the eye images to the mobile terminal, which will forward them to the cloud server. The cloud server will call the AI ​​model to identify the user's emotions, and output the corresponding emotional counseling solution as the image processing result when the user is depressed or tired. The cloud server sends the image processing results to the mobile terminal, and the mobile terminal forwards the image processing results to the smart glasses, so that the user can get the corresponding emotional counseling results on the smart glasses.

[0175] In addition to the image processing tasks in the above examples, smart glasses can also realize other functions such as physiological parameter detection, voice question and answer, UV reminder, music playback, making and receiving calls, etc. Those skilled in the art can undoubtedly understand and fully realize them based on the aforementioned implementation methods, and this manual will not go into details.

[0176] In some embodiments, after receiving the image processing result sent by the mobile terminal, the smart glasses can directly play the image processing result using a speaker, so that the user can hear the corresponding audio information. In other embodiments, the smart glasses can display the image processing result on a display device, so that the user can see the corresponding image information.

[0177] From the above, it can be seen that in the implementation mode of this specification, through the interaction between smart glasses and mobile terminals, combined with eye tracking, AI recognition and intelligent suggestions for objects that the user is looking at can be realized, thereby enriching the intelligent needs of smart glasses. In addition, by combining high-definition local images with low-definition panoramic images, the amount of communication data with the mobile terminal can be reduced, the transmission rate between the smart glasses and the mobile terminal can be increased, and the user waiting time can be reduced. At the same time, the contextual background information of the identified object can be fully combined to improve the accuracy of AI tasks, meet the needs of more tasks, and take into account both transmission efficiency and task effects.

[0178] In some embodiments, this specification provides an image processing device, which can be applied to the aforementioned smart glasses, such as Fig.11 As shown, the device comprises:

[0179] The image acquisition module 10 is configured to acquire a first image within the user's visual field and a gaze point of the user's eyes in response to detecting that the eye movement type of the user is non-jumping;

[0180] An image segmentation module 20 is configured to determine a target area on the first image based on the gaze point, and segment the target area to obtain a target area image;

[0181] A downsampling module 30 is configured to perform downsampling processing on the first image to obtain a second image;

[0182] The data communication module 40 is configured to send the target area image and the second image to a mobile terminal bound to the smart glasses, and receive an image processing result sent by the mobile terminal.

[0183] In some embodiments, the eye movement type includes beating and non-beating, and the image acquisition module 10 is configured to:

[0184] Collecting eye movement parameters of the user's eyes through an eye tracking device, wherein the eye movement parameters represent the eye movement speed of the user's eyes;

[0185] When the eye movement parameter is greater than or equal to a preset threshold, determining the eye movement type is a jump;

[0186] When the eye movement parameter is less than a preset threshold, the eye movement type is determined to be non-jumping.

[0187] In some embodiments, the image acquisition module 10 is configured to:

[0188] Using a light source assembly of an eye tracking device to emit an infrared light source to the user's eyes, and acquiring a light spot image of the user's eyes captured by an infrared camera of the eye tracking device, wherein the light spot image includes a light spot formed on the user's eyes by the infrared light source emitted by the light source assembly;

[0189] Performing image detection on the light spot image to determine the relative position of the pupil center of the user's eye and the light spot;

[0190] The gaze point of the user's eyes is determined based on the relative position and the pre-calibrated light spot position.

[0191] In some embodiments, the image segmentation module 20 is configured to:

[0192] Determine the image position of the gaze point on the first image based on a pre-calibrated mapping relationship, wherein the mapping relationship represents a mapping relationship between a coordinate system where the gaze point is located and an image coordinate system of the first image;

[0193] Performing image detection on the first image, and determining an image area including the image position where the gaze point is located as the target area;

[0194] Performing image segmentation on the target area to obtain the target area image.

[0195] In some embodiments, the present specification provides a storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in any of the above embodiments.

[0196] In some embodiments, the present specification provides a computer program product, which implements the method described in any of the above embodiments when executed by a computer.

Claims

1. A pair of smart glasses, characterized in that: include: An eye tracking device for collecting the gaze point of the user's eyes; A first camera is used to capture a first image, where the first image represents an image of a scene within a user's field of view; The processor is configured to, in response to detecting that the user's eye movement type is non-jumping, obtain a target area image from the segmentation of the first image based on the gaze point, downsample the first image to obtain a second image, send the target area image and the second image to a mobile terminal, and receive the image processing result sent by the mobile terminal.

2. The smart glasses according to claim 1, characterized in that: The eye tracking device includes a light source assembly and an infrared camera, wherein the light source assembly is used to emit an infrared light source toward the user's eyes, and the infrared camera is used to collect a light spot image of the user's eyes, wherein the light spot image includes a light spot formed on the user's eyes by the infrared light source emitted by the light source assembly; The processor is configured to perform image detection on the light spot image, determine the relative position between the pupil center of the user's eye and the light spot, and determine the gaze point of the user's eye based on the relative position and a pre-calibrated light spot position.

3. The smart glasses according to claim 2, characterized in that: The infrared camera is also used to collect eye movement parameters of the user's eyes, where the eye movement parameters represent the eye movement speed of the user's eyes; The processor is configured to receive the eye movement parameters collected by the infrared camera, and determine the eye movement type of the user's eyes based on the eye movement parameters.

4. The smart glasses according to claim 1, characterized in that: The smart glasses include a body, the body includes a lens frame and temples connected to both sides of the lens frame, the lens frame includes an inner wall facing the user's eyes and an outer wall away from the user's eyes; The eye tracking device is arranged on the inner wall of the lens frame, and the first camera is arranged on the outer wall of the lens frame.

5. The smart glasses according to any one of claims 1 to 4, characterized in that: Also comprising a speaker, the processor is configured to generate a voice instruction based on the image processing result, and the speaker is used to play the voice instruction; and / or, It also includes a display device, which is used to display the image processing result.

6. An image processing method, characterized in that: Applied to smart glasses, the method comprises: In response to detecting that the eye movement type of the user is non-jumping, acquiring a first image within the user's visual field and a gaze point of the user's eyes; Determine a target area on the first image based on the gaze point, and segment the target area to obtain a target area image; Downsampling the first image to obtain a second image; The target area image and the second image are sent to a mobile terminal bound to the smart glasses, and an image processing result sent by the mobile terminal is received.

7. The method according to claim 6, characterized in that The eye movement type includes beating and non-beating, and the process of detecting the user's eye movement type includes: Collecting eye movement parameters of the user's eyes through an eye tracking device, wherein the eye movement parameters represent the eye movement speed of the user's eyes; When the eye movement parameter is greater than or equal to a preset threshold, determining the eye movement type is a jump; When the eye movement parameter is less than a preset threshold, the eye movement type is determined to be non-jumping.

8. The method according to claim 6, characterized in that The process of obtaining the user's eye gaze point includes: Using a light source assembly of an eye tracking device to emit an infrared light source to the user's eyes, and acquiring a light spot image of the user's eyes captured by an infrared camera of the eye tracking device, wherein the light spot image includes a light spot formed on the user's eyes by the infrared light source emitted by the light source assembly; Performing image detection on the light spot image to determine the relative position of the pupil center of the user's eye and the light spot; The gaze point of the user's eyes is determined based on the relative position and the pre-calibrated light spot position.

9. The method according to claim 6, characterized in that The determining a target area on the first image based on the gaze point, and segmenting the target area to obtain a target area image, comprises: Determine the image position of the gaze point on the first image based on a pre-calibrated mapping relationship, wherein the mapping relationship represents a mapping relationship between a coordinate system where the gaze point is located and an image coordinate system of the first image; Performing image detection on the first image, and determining an image area including the image position where the gaze point is located as the target area; Performing image segmentation on the target area to obtain the target area image.

10. The method according to claim 6, characterized in that After receiving the image processing result sent by the mobile terminal, the method further includes: generating a voice command based on the image processing result, and playing the voice command; and / or, The image processing result is displayed on a display device of the smart glasses.

11. An image processing device, characterized in that: Applied to smart glasses, the device comprises: An image acquisition module, configured to acquire a first image within the user's visual field and a gaze point of the user's eyes in response to detecting that the user's eye movement type is non-jumping; an image segmentation module, configured to determine a target area on the first image based on the gaze point, and segment the target area to obtain a target area image; A downsampling module, configured to perform downsampling processing on the first image to obtain a second image; The data communication module is configured to send the target area image and the second image to a mobile terminal bound to the smart glasses, and receive an image processing result sent by the mobile terminal.

12. A storage medium, characterized in that: Computer instructions are stored, and the computer instructions are used to make a computer execute the method according to any one of claims 6 to 10.

13. A computer program product, characterized in that When the computer program product is executed by a computer, the method according to any one of claims 6 to 10 is implemented.

Images