Human-computer interaction method based on smart glasses, smart glasses and readable medium

Abstract

The embodiments of the present disclosure disclose a human-computer interaction method based on smart glasses, smart glasses, and readable media. A specific implementation of the method includes: acquiring scene visual data and eye movement data; generating eye movement information; generating an image area to be processed; generating a corresponding image processing result; performing the following steps: generating corresponding first human-computer interaction information; performing a human-computer interaction task corresponding to the first human-computer interaction information; in response to detecting voice information, performing the following steps: obtaining voice content information corresponding to the voice information; generating corresponding second human-computer interaction information; performing a human-computer interaction task corresponding to the second human-computer interaction information. This implementation can reduce the waste of computer resources during image processing and improve the processing speed during human-computer interaction.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a human-computer interaction method based on smart glasses, smart glasses, and readable media. Background Art

[0002] Human-computer interaction based on smart glasses is a technology that uses smart glasses as an interactive medium to exchange information and control operations with users. At present, when performing human-computer interaction through smart glasses, the method usually adopted is: first, the user's eye movement data is processed through an eye tracking algorithm to obtain eye movement data. Then, each object included in the scene image is identified to obtain each recognition result. Then, the recognition result corresponding to the eye movement data in each recognition result is determined as human-computer interaction information. Finally, the human-computer interaction information is rendered and displayed in a specific area to perform human-computer interaction.

[0003] However, when the above method is used for human-computer interaction, the following technical problems often occur:

[0004] First, when recognizing an image, it is necessary to first recognize all objects in the image, and then feed back the recognition results corresponding to the user's needs to the user, resulting in too many useless recognition results, which in turn leads to a waste of computing resources during image processing and a poor processing speed during human-computer interaction.

[0005] Second, as the number of interactions increases, the reliability of the eye tracking algorithm gradually decreases, resulting in a gradual decrease in the accuracy of the eye movement data generated by the eye tracking algorithm, which in turn leads to a low reliability of the human-computer interaction results determined using the eye movement data, resulting in a high error rate in the execution of human-computer interaction tasks and a waste of computing resources when executing human-computer interaction tasks.

[0006] Third, after the rendered human-computer interaction information is displayed in a specific area, the user needs to actively read and understand it. Moreover, the displayed information is easily disturbed by visual elements such as advertisements and icons, which distracts the user and makes it difficult to focus on the main information, resulting in low efficiency in receiving information. At the same time, the human-computer interaction information is easily disturbed by factors such as light, fonts, and display screen resolution during the display process, resulting in low accuracy of information transmission. Summary of the invention

[0007] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0008] Some embodiments of the present disclosure propose a human-computer interaction method based on smart glasses, smart glasses, and computer-readable media to solve one or more of the technical problems mentioned in the above background technology section.

[0009] In a first aspect, some embodiments of the present disclosure provide a human-computer interaction method based on smart glasses, the method comprising: acquiring scene visual data and eye movement data; generating eye movement information based on the eye movement data, wherein the eye movement information comprises eye movement trajectory information, gaze point position and eye movement pattern information; generating an image area to be processed based on the eye movement information, the scene visual data and each previously acquired historical image area to be processed; generating a corresponding image processing result based on the image area to be processed; in response to determining that the image processing result satisfies a preset result condition, performing the following steps: generating corresponding first human-computer interaction information based on the image processing result, a preset call information group and a preset prompt information group; performing a human-computer interaction task corresponding to the first human-computer interaction information based on the first human-computer interaction information; in response to detecting voice information, performing the following steps: performing data conversion processing on the voice information to obtain voice content information corresponding to the voice information; generating corresponding second human-computer interaction information based on the image processing result, the voice content information, the call information group and the prompt information group; performing a human-computer interaction task corresponding to the second human-computer interaction information based on the second human-computer interaction information.

[0010] In a second aspect, some embodiments of the present disclosure provide a pair of smart glasses, comprising: a first visual sensor configured to acquire scene visual data; a second visual sensor configured to acquire eye movement data; a sound collector configured to detect voice information; one or more processors; and a storage device storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0011] In a third aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0012] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the human-computer interaction method based on smart glasses in some embodiments of the present disclosure, the waste of computing resources during image processing can be reduced, and the processing speed during human-computer interaction can be improved. Specifically, the reason for the waste of computing resources during image processing and the slow processing speed during human-computer interaction is that when recognizing an image, it is necessary to first recognize all objects in the image, and then feed back the recognition results corresponding to the user's needs in each recognition result to the user, resulting in too many useless recognition results, which in turn leads to waste of computing resources during image processing and poor processing speed during human-computer interaction. Based on this, the human-computer interaction method based on smart glasses in some embodiments of the present disclosure first obtains scene visual data and eye movement data. Thus, the original data of human-computer interaction can be obtained. Secondly, based on the above eye movement data, eye movement information is generated, wherein the above eye movement information includes eye movement trajectory information, gaze point position and eye movement pattern information. Thus, the user's eye movement information can be obtained through the user's eye movement data. Then, based on the above eye movement information, the above scene visual data and each previously acquired historical image area to be processed, the image area to be processed is generated. Thus, the image area that the user is looking at can be determined through the user's eye movement information, and only the determined image area can be identified in the subsequent processing, which can reduce useless identification results, improve the processing speed of image processing, and further reduce the waste of computing resources during image processing, and improve the processing speed of human-computer interaction. Then, based on the above-mentioned image area to be processed, the corresponding image processing result is generated. Thus, the information corresponding to the image area that the user is looking at can be obtained. Then, in response to determining that the above-mentioned image processing result meets the preset result condition, the following steps are performed: first, based on the above-mentioned image processing result, the preset call information group and the preset prompt information group, the corresponding first human-computer interaction information is generated. Thus, the human-computer interaction information corresponding to the user's needs can be quickly generated through the above-mentioned image processing result, the above-mentioned call information group and the above-mentioned prompt information group. Then, based on the above-mentioned first human-computer interaction information, the human-computer interaction task corresponding to the above-mentioned first human-computer interaction information is executed. Thus, the function corresponding to the user's needs can be quickly called through the first human-computer interaction information, the human-computer interaction task corresponding to the user's needs can be quickly executed, and the processing speed during human-computer interaction can be greatly improved. Then, in response to detecting the voice information, the following steps are performed: first, the voice information is subjected to data conversion processing to obtain the voice content information corresponding to the voice information. Thus, the user's voice information can be converted into text information. Then, based on the image processing result, the voice content information, the call information group and the prompt information group, the corresponding second human-computer interaction information is generated. Thus, the human-computer interaction information corresponding to the user's needs can be obtained. Finally, based on the second human-computer interaction information, the human-computer interaction task corresponding to the second human-computer interaction information is executed.Therefore, the human-computer interaction tasks corresponding to the user's needs can be quickly executed through the human-computer interaction information, which can greatly improve the processing speed of human-computer interaction. Also, because the image area that needs to be identified can be determined based on the user's eye movement information, and then the image area is identified, without the need to identify all image areas, it is possible to reduce useless recognition results and reduce the waste of computing resources during image processing, thereby improving the processing speed of human-computer interaction. Also, because the human-computer interaction information corresponding to the user's needs can be generated by combining the callable information group and the prompt information group, and then the human-computer interaction tasks are executed according to the human-computer interaction information, it is possible to directly determine the operation command to be executed through the generated human-computer interaction information, which can improve the processing speed of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flow chart of some embodiments of the human-computer interaction method based on smart glasses according to the present disclosure;

[0015] Figure 2 It is a schematic diagram of the structure of smart glasses suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0018] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0019] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0020] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0021] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0022] Figure 1 The process 100 of some embodiments of the human-computer interaction method based on smart glasses according to the present disclosure is shown. The human-computer interaction method based on smart glasses includes the following steps:

[0023] Step 101, obtaining scene visual data and eye movement data.

[0024] In some embodiments, the execution subject (e.g., a computing device) of the human-computer interaction method based on smart glasses can obtain scene visual data and eye movement data. Among them, the execution subject can be smart glasses. The scene visual data can be video data corresponding to the scene that the user is looking at at the current moment. The scene visual data includes various scene video images. The scene video image can be a video image included in the scene visual data. The eye movement data can be video data corresponding to the user's eye rotation. The eye movement data can include left eye eye movement data and right eye eye movement data. The left eye eye movement data can be video data corresponding to the user's left eye rotation. The right eye eye movement data can be video data corresponding to the user's right eye rotation. The left eye eye movement data can include various left eye video images. The right eye eye movement data can include various right eye video images. The left eye video image can be a video image corresponding to the left eye eye movement data. The right eye video image can be a video image corresponding to the right eye eye movement data. Each of the left eye video images corresponds to a pre-identified left eye pupil position. Each of the above-mentioned right-eye video images corresponds to a pre-identified right-eye pupil position. The above-mentioned left-eye pupil position can be the two-dimensional coordinates of the pupil of the user's left eye in the corresponding left-eye video image. In practice, for each of the above-mentioned left-eye video images, the above-mentioned execution subject can locate the center position of the left-eye pupil in the above-mentioned left-eye video image through the eyeball center positioning and tracking algorithm. Then, the above-mentioned execution subject can use the first pixel point in the upper left corner of the above-mentioned left-eye video image as the coordinate origin, the horizontal right direction of the above-mentioned left-eye video image as the horizontal axis, and the vertical downward direction of the above-mentioned left-eye video image as the vertical axis to construct a coordinate system as the left-eye coordinate system, and then obtain the left-eye pupil position corresponding to the above-mentioned left-eye video image in the above-mentioned left-eye coordinate system. The above-mentioned right-eye pupil position can be the two-dimensional coordinates of the pupil of the user's right eye in the corresponding right-eye video image. The method of obtaining the right-eye pupil position is similar to that of the above-mentioned left-eye pupil position, which will not be repeated here. In practice, the above-mentioned execution subject can obtain scene visual data through the first visual sensor. Among them, the above-mentioned first visual sensor can be a visual sensor for obtaining scene visual data. The above-mentioned execution subject can include two second visual sensors. In practice, the above-mentioned execution subject can obtain left eye eye movement data through one of the second visual sensors. Then, the right eye eye movement data can be obtained through another second visual sensor. Finally, the above-mentioned left eye eye movement data and the above-mentioned right eye eye movement data can be combined into eye movement data. The above-mentioned second visual sensor can be a visual sensor for obtaining eye movement data. The above-mentioned visual sensor can be a device for obtaining image information or video information corresponding to the external environment. For example, the above-mentioned visual sensor can be a smart camera or an infrared camera.

[0025] Step 102: Generate eye movement information based on the eye movement data.

[0026] In some embodiments, the execution subject may generate eye movement information based on the eye movement data. The eye movement information may be information corresponding to the user's eye rotation. The eye movement information may include eye movement trajectory information, gaze point position and eye movement pattern information. The eye movement trajectory information may be trajectory information of the user's line of sight moving in the scene visual data. The coordinate system corresponding to the trajectory information is a pre-constructed world coordinate system. The gaze point position may be a three-dimensional coordinate corresponding to the position where the user is gazing in the scene visual data. The coordinate system corresponding to the gaze point position is the world coordinate system. The eye movement pattern information may be information for characterizing the speed of the user's eye rotation, the amplitude of the eye twitching, the duration of gazing at a certain position and the gaze direction. In practice, first, the execution subject may generate a stereo model corresponding to the current scene through the SLAM algorithm (Simultaneous Localization And Mapping) and the scene visual data. Then, a preset point in the stereo model is determined as the coordinate origin, and a coordinate system is constructed as a world coordinate system with the horizontal eastward direction as the horizontal axis, the horizontal southward direction as the vertical axis, and the vertical upward direction as the vertical axis.

[0027] In some optional implementations of some embodiments, the eye movement data may include left eye movement data and right eye movement data. The execution subject may generate eye movement information based on the eye movement data by the following steps:

[0028] The first step is to preprocess the left eye movement data and the right eye movement data included in the eye movement data to obtain preprocessed eye movement data. The preprocessed eye movement data may be video data of the user's eye rotation after preprocessing. The preprocessed eye movement data includes each preprocessed left eye pupil position and each preprocessed right eye pupil position. The preprocessed left eye pupil position may be the corresponding left eye pupil position in the preprocessed left eye eye movement data. The preprocessed right eye pupil position may be the corresponding right eye pupil position in the preprocessed right eye eye movement data. In practice, the execution subject may denoise, filter and adjust the resolution of the eye movement data to obtain preprocessed eye movement data.

[0029] The second step is to generate each gaze point coordinate based on each pre-processed left eye pupil position and each pre-processed right eye pupil position included in the above-mentioned pre-processed eye movement data. Among them, the above-mentioned gaze point coordinates can be three-dimensional coordinates used to characterize the user's gaze position in the above-mentioned world coordinate system. Each of the above-mentioned gaze point coordinates corresponds to a generation time and an end time. The above-mentioned generation time can be a time point when the above-mentioned gaze point coordinates are generated. The above-mentioned end time can be a time point when the next gaze point coordinate close to the above-mentioned gaze point coordinates is generated. In practice, first, the above-mentioned execution subject can perform coordinate conversion on each pre-processed left eye pupil position and each pre-processed right eye pupil position included in the above-mentioned pre-processed eye movement data through pre-stored calibration data. Among them, the above-mentioned calibration data can include a first internal reference matrix, a second internal reference matrix, a first external reference matrix, and a second external reference matrix. The above-mentioned first internal reference matrix can be an internal reference matrix corresponding to the second visual sensor used to obtain left eye eye movement data. The above-mentioned first external reference matrix can be an external reference matrix corresponding to the second visual sensor used to obtain left eye eye movement data. The second internal reference matrix may be an internal reference matrix corresponding to the second visual sensor for obtaining right eye eye movement data. The second external reference matrix may be an external reference matrix corresponding to the second visual sensor for obtaining right eye eye movement data. In practice, first, for each of the pre-processed left eye pupil positions, the execution subject may convert the pre-processed left eye pupil position into a corresponding matrix as a left eye pixel matrix. For example, when the left eye pupil position is (1,3), the corresponding left eye pixel matrix is ​​the transpose of the (1,3,1) matrix. Then, the product of the first internal reference matrix, the first external reference matrix, and the left eye pixel matrix may be determined as the left eye gaze point coordinates. Thus, the left eye gaze point coordinates may be obtained. The method of obtaining the right eye gaze point coordinates is similar to the method of obtaining the left eye gaze point coordinates, which will not be repeated here. Then, for each of the left eye gaze point coordinates, the execution subject may determine the right eye gaze point coordinate corresponding to the left eye gaze point coordinate as the gaze point coordinate to be fused. Then, the left eye gaze point coordinates and the average value of the gaze point coordinates to be fused can be determined as the corresponding gaze point coordinates. As an example: when the left eye gaze point coordinates are (1, 2) and the right eye gaze point coordinates are (1, 4), the gaze point coordinates can be (1, 3).

[0030] The third step is to construct eye movement trajectory information based on the above-mentioned gaze point coordinates and the generation times corresponding to the above-mentioned gaze point coordinates. The above-mentioned eye movement trajectory information can represent the trajectory of the user's eye movement. In practice, first, the above-mentioned execution subject can generate a curve corresponding to each gaze point coordinate by the least squares method. Then, the generated curve can be determined as the above-mentioned eye movement trajectory information.

[0031] The fourth step is to generate the gaze duration corresponding to each of the above-mentioned gaze point coordinates based on the generation time and end time corresponding to the above-mentioned gaze point coordinate. The above-mentioned gaze duration can represent the length of time of continuous gaze at a position. In practice, the above-mentioned execution entity can determine the difference between the generation time and the end time as the gaze duration. For example, when the above-mentioned generation time is 9 hours, 10 minutes, 5 seconds and 10 milliseconds, and the end time is 9 hours, 10 minutes, 5 seconds and 55 milliseconds, the gaze duration is 45 milliseconds.

[0032] Step 5: For every two gaze point coordinates that meet the preset generation time conditions among the above gaze point coordinates, perform the following steps:

[0033] The first sub-step is to generate the corresponding eye saccade amplitude based on the above two fixation point coordinates. The above generation time condition may be that the generation time corresponding to the two fixation point coordinates is continuous. The above eye saccade amplitude may be the distance between the above two fixation point coordinates. In practice, the above execution subject may obtain the eye saccade amplitude between the two fixation point coordinates by the distance formula between two points.

[0034] The second sub-step is to generate corresponding gaze direction information based on the above two gaze point coordinates. The above gaze direction information can represent the direction of movement of the user's line of sight. In practice, the above execution subject can connect the above two gaze point coordinates by the generation time corresponding to the above two gaze point coordinates, with the direction from the gaze point coordinate with an earlier generation time to the gaze point coordinate with a later generation time, to obtain a directed line segment. Then, the direction corresponding to the directed line segment can be determined as the above gaze direction information.

[0035] In the sixth step, the generated fixation durations, the generated eye saccade amplitudes, and the generated fixation direction information are integrated to obtain eye movement pattern information. In practice, the above-mentioned execution subject can combine the generated fixation durations, the generated eye saccade amplitudes, and the generated fixation direction information into eye movement pattern information.

[0036] In the seventh step, the gaze point coordinates satisfying the preset gaze condition among the above-mentioned gaze point coordinates are determined as the gaze point position. The above-mentioned preset gaze condition may be that the gaze point coordinates correspond to the longest gaze duration.

[0037] In step 8, the gaze point position, the eye movement trajectory information and the eye movement pattern information are integrated to obtain eye movement information. In practice, the execution subject may combine the gaze point position, the eye movement trajectory information and the eye movement pattern information into the eye movement information.

[0038] Step 103, generating an image region to be processed based on the eye movement information, the scene visual data and each previously acquired historical image region to be processed.

[0039] In some embodiments, the execution subject may generate an image region to be processed based on the eye movement information, the scene visual data and various previously acquired historical image regions to be processed.

[0040] In some optional implementations of some embodiments, the execution subject may generate the image region to be processed based on the eye movement information, the scene visual data and each previously acquired historical image region to be processed by the following steps:

[0041] The first step is, in response to determining that the gaze point position included in the above eye movement information meets the preset position condition, determining the historical image area to be processed that meets the preset area condition in the above various historical image areas to be processed as the image area to be processed. Among them, the image area to be processed may be an image area that needs to be recognized. The above preset position condition may be that the gaze point position is within the range included in the historical image area to be processed. The above historical image area to be processed may be the image area to be processed stored during the last human-computer interaction. The above preset area condition may be that the historical image area to be processed contains the above gaze point position.

[0042] In the second step, in response to determining that the gaze point position included in the above eye movement information does not satisfy the above preset position condition, based on the above gaze point position and the above scene visual data, an image area of ​​a preset shape is determined as the image area to be processed. The size of the above image area to be processed is a preset size. In practice, the above execution subject may determine the scene video image that satisfies the preset scene condition among the various scene video images included in the above scene visual data as the scene image. Then, the image area of ​​a preset shape and preset size in the above scene image may be determined as the image area to be processed with the gaze point position as the center. The preset shape may be a preset image shape. The preset size may be a preset image size. The preset scene condition may be that the scene video image contains the above gaze point position and the corresponding video time point is the earliest.

[0043] In some optional implementations of some embodiments, the execution subject may determine an image region of a preset shape as the image region to be processed based on the gaze point position and the scene visual data. The method for determining the image region to be processed may refer to the specific implementation in step 103, which will not be described in detail here.

[0044] Step 104: Generate a corresponding image processing result based on the image area to be processed.

[0045] In some embodiments, the execution subject may generate a corresponding image processing result based on the image area to be processed. The image processing result may be information corresponding to an object or a person in the image area to be processed. In practice, first, the execution subject may segment the object in the image area to be processed by using a segmentation model (SAM). Then, the segmented object may be identified by YOLO (You Only Look Once) to obtain the name corresponding to the object. Then, the execution subject may search the identified name through a search engine to obtain object information corresponding to the object in the image. The object information may include but is not limited to at least one of the following: the name, category, and size corresponding to the object. Finally, the execution subject may determine the information corresponding to the object as the image processing result. In practice, the execution subject may also directly input the image area to be processed and the preset output prompt information into the multimodal large model to obtain the corresponding image processing result. In practice, when the image area to be processed contains text information, the execution subject may also identify the corresponding content through optical character recognition technology and add the identified content to the preset output prompt information. Then, the image area to be processed and the output prompt information are input into the multimodal large model. The multimodal model may be a model that takes the image and the output prompt information as input and outputs the text corresponding to the output prompt information. For example, the multimodal model may be GPT-4. The output prompt information may be text information for guiding the output result. For example, the output prompt information may be “Please identify the person in the image and give his name and the last time you met him”.

[0046] Optionally, before step 105, the execution subject may further perform the following steps:

[0047] The first step is to generate eye movement operation information based on the eye movement trajectory information and eye movement pattern information included in the eye movement information. The eye movement operation information can be the name of the eye movement behavior corresponding to the user's eye jump. The eye movement operation information can include but is not limited to: blinking twice in a row, closing the left eye, and closing the right eye. In practice, the execution subject can identify and process the eye movement trajectory information and the eye movement pattern information by combining the programming language and the open source computer vision function library to obtain the eye movement operation information. The open source computer vision function library can be opencv.

[0048] The second step is to identify and process the above-mentioned eye movement operation information to obtain the eye movement operation recognition result corresponding to the above-mentioned eye movement operation information. Among them, the above-mentioned eye movement operation recognition result can be a label used to characterize the above-mentioned eye movement operation information. For example, when the above-mentioned eye movement operation information is "blinking twice in a row", the corresponding eye movement operation recognition result can be "blinking continuously". In practice, for each eye movement category in the preset eye movement category group, the above-mentioned execution subject can generate the probability that the above-mentioned eye movement operation information belongs to the above-mentioned eye movement category through a text classification algorithm. Then, the eye movement category with the highest corresponding probability can be determined as the eye movement operation recognition result corresponding to the above-mentioned eye movement information. Among them, the eye movement category in the above-mentioned eye movement category group can be a label used to characterize eye movement behavior. For example, the above-mentioned eye movement category can be "blinking continuously" or "left eye closed". The above-mentioned text classification algorithm can be a naive Bayes algorithm.

[0049] In the third step, in response to determining that the eye movement operation recognition result meets the preset operation condition, the following steps are performed:

[0050] The first sub-step is to determine the callable information corresponding to the eye movement operation recognition result based on the preset callable information mapping relationship. Among them, the preset operation condition can be that the eye movement operation recognition result belongs to the eye movement operation function information. The eye movement operation function information can be an eye movement behavior that can trigger the function corresponding to the glasses. For example, the eye movement operation recognition information can be but not limited to: continuous blinking, left eye closure, right eye closure. The callable information mapping relationship can be used to characterize the correspondence between the eye movement operation recognition result and the callable information. For example, the callable information mapping relationship can be "continuous blinking corresponds to taking pictures", "left eye closure corresponds to starting recording", "right eye closure corresponds to pausing recording". The callable information can be the information corresponding to the custom function that can be called. The callable information can include the name and parameters corresponding to the function. For example, the callable information can be makecall (123456), sendsms (123, "information"). The custom function can be a function pre-set by the target technician through code to implement the corresponding function.

[0051] The second sub-step is to execute the corresponding human-computer interaction task based on the above-mentioned callable information and obtain the corresponding execution result. The human-computer interaction task may be an operation to be executed. The above-mentioned human-computer interaction task may be an operation corresponding to the above-mentioned callable information. The above-mentioned execution result may be a label used to characterize whether the above-mentioned human-computer interaction task is successfully executed. For example, the above-mentioned execution result may be "execution successful" or "execution failed".

[0052] The third sub-step is to perform voice conversion processing on the above-mentioned execution result to obtain the execution result voice data corresponding to the above-mentioned execution result. The above-mentioned execution result voice data may be the voice corresponding to the above-mentioned execution result. In practice, the above-mentioned execution subject may convert the above-mentioned execution result into the corresponding execution result voice data through TTS technology (Text-To-Speech, from text to speech).

[0053] The fourth sub-step is to play the voice data of the above execution result.

[0054] Step 105, in response to determining that the image processing result meets the preset result condition, executing the following steps:

[0055] Step 1051: Generate corresponding first human-computer interaction information based on the image processing result, the preset calling information group and the preset prompt information group.

[0056] In some embodiments, the execution subject may generate corresponding first human-computer interaction information based on the image processing result, a preset call information group and a preset prompt information group.

[0057] In some optional implementations of some embodiments, the execution subject may generate corresponding first human-computer interaction information based on the image processing result, the preset call information group and the preset prompt information group through the following steps:

[0058] The first step is to generate corresponding prompt result information based on the above image processing result, the preset call information group and the preset prompt information group. Among them, the call information in the above call information group can be information corresponding to a preset custom function. The above call information can include the name and parameters corresponding to the function. For example, the above call information can be FaceRecognition(), AttentionLevelEstimation(), FatigueDetection(), CognitiveDisorderDetection(). The prompt information in the above prompt information group can be text information for guiding the output result. For example, the above prompt information can be "refer to the call information group, complete the interaction with the user according to the image processing result, and output the result in json format". The above prompt result information can be the output result obtained after the above image processing result, the above call information group and one of the prompt information in the above prompt information group are input into the large language model. The above large language model can be a model that takes a text sequence as input and outputs the text corresponding to the prompt information in the text sequence. For example, the above large language model can be GPT-4. The above prompt result information corresponds to an information type. The above information type can be text information or function information. The text information may be prompt result information that does not contain any of the call information in the call information group. The function information may be prompt result information that contains any of the call information in the call information group. In practice, first, the execution subject may determine the prompt information in the prompt information group corresponding to the input image processing result as the prompt information to be input. Then, the image processing result, the call information group and the prompt information to be input may be input into the large language model to obtain the prompt result information.

[0059] In the second step, in response to determining that the prompt result information satisfies a preset information condition, the prompt result information is determined as the first human-computer interaction information. The preset information condition may be that the prompt result information is not a null value.

[0060] The third step is to clear the prompt result information in response to determining that the prompt result information does not meet the preset information condition, and generate corresponding clear prompt information. The clear prompt information can be a label used to indicate that the prompt result information has been cleared. For example, the clear prompt information can be "cleared". In practice, the execution subject can clear the prompt result information through an operation command. The execution subject can generate the clear prompt information through a programming language.

[0061] Step 1052: Based on the first human-computer interaction information, execute the human-computer interaction task corresponding to the first human-computer interaction information.

[0062] In some embodiments, the execution subject may execute the human-computer interaction task corresponding to the first human-computer interaction information based on the first human-computer interaction information.

[0063] In some optional implementations of some embodiments, the first human-computer interaction information corresponds to a human-computer interaction type. The execution subject may perform the human-computer interaction task corresponding to the first human-computer interaction information based on the first human-computer interaction information by the following steps:

[0064] In the first step, in response to determining that the human-computer interaction type corresponding to the first human-computer interaction information meets a first preset type condition, the following steps are performed:

[0065] The first sub-step is to perform speech conversion processing on the above-mentioned first human-computer interaction information to obtain the interactive information audio data corresponding to the above-mentioned human-computer interaction information. Among them, the above-mentioned human-computer interaction type can characterize the type corresponding to the above-mentioned first human-computer interaction information. The above-mentioned human-computer interaction type includes text information and function information. The above-mentioned first preset type condition can be that the above-mentioned first human-computer interaction information is text information. The above-mentioned interactive information audio data can be the audio data corresponding to the above-mentioned first human-computer interaction information. In practice, the above-mentioned execution subject can perform speech conversion processing on the above-mentioned first human-computer interaction information using the above-mentioned TTS technology to obtain the interactive information audio data.

[0066] The second sub-step is to play the above-mentioned interactive information audio data.

[0067] In the second step, in response to determining that the interaction information type corresponding to the first human-computer interaction information meets the second preset type condition, the following steps are performed:

[0068] In the first sub-step, the first human-computer interaction information is parsed and processed to obtain corresponding call information. Among them, the second preset type condition may be that the second human-computer interaction information belongs to function type information. The call information may be information corresponding to the function included in the first human-computer interaction information. The call information includes the name and parameters corresponding to the function. In practice, the execution subject may parse the first human-computer interaction information through a custom function and extract the corresponding call information. Among them, the custom function may be SelectFunction().

[0069] The second sub-step is to determine the corresponding human-computer interaction task based on the above call information. The human-computer interaction task may be the operation to be performed. The above human-computer interaction task may be the operation corresponding to the above call information. As an example: when the above call information is takephoto(), the corresponding human-computer interaction task is to take a photo. When the above call information is makecall(1234567), the corresponding human-computer interaction task is to call the terminal with the number 1234567.

[0070] The third sub-step is to execute the human-computer interaction task to obtain a corresponding human-computer interaction execution result. The human-computer interaction execution result may be a label used to indicate whether the human-computer interaction task is successfully executed. For example, the human-computer interaction execution result may be "execution successful" or "execution failed".

[0071] The fourth sub-step is to perform voice conversion processing on the above-mentioned human-computer interaction execution result to obtain corresponding execution result audio data. The above-mentioned execution result audio data may be audio data corresponding to the above-mentioned human-computer interaction execution result. In practice, the above-mentioned execution subject may convert the above-mentioned human-computer interaction execution result into the above-mentioned execution result audio data by TTS technology.

[0072] The fifth sub-step is to play the audio data of the above execution result.

[0073] The above technical solution and its related contents, as an inventive point of the embodiment of the present disclosure, solve the technical problem three mentioned in the background technology: "After the human-computer interaction information after rendering is displayed in a specific area, the user needs to actively read and understand it. And the displayed information is easily disturbed by visual elements such as advertisements and icons, which causes the user's attention to be distracted and it is difficult to focus on the main information, which leads to low efficiency of information reception by the user. At the same time, the human-computer interaction information is easily disturbed by factors such as light, fonts, and display screen resolution during the display process, resulting in low accuracy of information transmission." The factors that lead to low efficiency of information reception by the user and low accuracy of information transmission are often as follows: After the human-computer interaction information after rendering is displayed in a specific area, the user needs to actively read and understand it. And the displayed information is easily disturbed by visual elements such as advertisements and icons, which causes the user's attention to be distracted and it is difficult to focus on the main information, which leads to low efficiency of information reception by the user. At the same time, the human-computer interaction information is easily disturbed by factors such as light, fonts, and display screen resolution during the display process, resulting in low accuracy of information transmission. If the above factors are solved, the user's efficiency of receiving information can be improved and the accuracy of information transmission can be improved. In order to achieve this effect, the present disclosure performs the following steps in response to determining that the human-computer interaction type corresponding to the above-mentioned first human-computer interaction information meets the first preset type condition: first, perform voice conversion processing on the above-mentioned first human-computer interaction information to obtain the interactive information audio data corresponding to the above-mentioned human-computer interaction information. Thus, the human-computer interaction information can be converted into voice data. Then, the above-mentioned interactive information audio data is played. Thus, the above-mentioned first human-computer interaction information can be transmitted to the user in the form of playing the audio data, thereby improving the user's efficiency in receiving information and improving the accuracy of information transmission. Then, in response to determining that the interactive information type corresponding to the above-mentioned first human-computer interaction information meets the second preset type condition, the following steps are performed: first, perform parsing processing on the above-mentioned first human-computer interaction information to obtain the corresponding call information. Thus, the function information to be called can be obtained. Then, based on the above-mentioned call information, the corresponding human-computer interaction task is determined. Thus, the corresponding human-computer interaction task can be executed according to the above-mentioned first human-computer interaction information. Then, the above-mentioned human-computer interaction task is executed to obtain the corresponding human-computer interaction execution result. Thus, the execution result of the human-computer interaction task can be obtained. Then, the above-mentioned human-computer interaction execution result is processed by voice conversion to obtain corresponding execution result audio data. In this way, the execution result of the human-computer interaction can be converted into corresponding audio data. Finally, the above-mentioned execution result audio data is played. In this way, the execution result can be delivered to the user. Also, because the human-computer interaction information can be converted into voice data and then the voice data is played to deliver information to the user, the user's reception efficiency of information can be improved, and the accuracy of information delivery can be improved.

[0074] Optionally, after step 1052, the execution entity may further perform the following steps:

[0075] The first step is to receive a self-calibration information group. The self-calibration information may be information corresponding to a condition that can trigger a self-calibration operation. Each self-calibration information in the self-calibration information group corresponds to a trigger condition. The self-calibration information may be, but is not limited to, time interval information, time point information, and user operation information. The time interval information may be the interval time from the last self-calibration. The trigger condition corresponding to the time interval information may be that the time interval information is equal to a preset time length. The time point information may be the time point corresponding to the current moment. The trigger condition corresponding to the time point information may be that the time point information is equal to a preset time point. The user operation information may be the name corresponding to the operation actively initiated by the user. For example, the user operation information may include, but is not limited to, taking a photo, recording a video. The trigger condition corresponding to the user operation information may be that the user operation information is a self-calibration operation. In practice, the execution subject may receive a self-calibration information group sent by an information storage server. The information storage server may be a server for storing and managing self-calibration information. The eye movement information is generated by a preset eye movement information generation algorithm. The eye movement information generation algorithm may be an algorithm that uses the eye movement data as input and the eye movement information as output. For example, the eye movement information generation algorithm may be a pupil cornea reflection vector method.

[0076] In the second step, in response to determining that the self-calibration information group meets the preset self-calibration condition, the following steps are performed:

[0077] The first sub-step is to obtain historical human-computer interaction data. Among them, the above-mentioned preset self-calibration condition can be that any self-calibration information in the above-mentioned self-calibration information group reaches the corresponding trigger condition. The above-mentioned historical human-computer interaction data can be the data stored by the above-mentioned execution subject when the user performs human-computer interaction before the current time point. The above-mentioned historical human-computer interaction data includes historical eye movement data and historical image processing results. The above-mentioned historical eye movement data can be the eye movement data stored when the above-mentioned execution subject interacts with the user within a preset time range. The above-mentioned historical eye movement data includes each historical pupil position. The above-mentioned historical pupil position can be the pupil position stored when the above-mentioned execution subject interacts with the user within the above-mentioned time range. The above-mentioned historical image processing result can be the image processing result stored when the above-mentioned execution subject interacts with the user within the above-mentioned time range. The above-mentioned time range can be from the time when the above-mentioned execution subject first interacts with the user to the time when the execution subject interacts with the user most recently from the current moment.

[0078] The second sub-step is to generate each historical fixation point position based on the above historical human-computer interaction data. Wherein, the above historical fixation point position can be the fixation point position stored when the human-computer interaction is performed before the current time point. In practice, first, the above execution subject can perform data extraction processing on the above historical human-computer interaction through a density-based clustering algorithm to obtain each historical pupil position. Then, the above execution subject generates each historical fixation point position through each historical pupil position. The method for generating each historical fixation point position can refer to the specific implementation method of step 102, which will not be repeated here.

[0079] In the third sub-step, for each of the above-mentioned historical fixation point positions, the following steps are performed:

[0080] Sub-step 1: Based on the above historical human-computer interaction data, generate the fixation point position to be calibrated corresponding to the above historical fixation point position. Wherein, the above fixation point position to be calibrated corresponds to a confidence level. The above fixation point position to be calibrated can be the fixation point coordinates obtained by the above eye movement information generation algorithm. The above confidence level can characterize the accuracy of the generated fixation point position to be calibrated.

[0081] Sub-step two, integrate the above-mentioned historical gaze point positions, the above-mentioned gaze point positions to be calibrated, and the confidences corresponding to the above-mentioned gaze point positions to be calibrated to obtain the gaze point data pair. In practice, the above-mentioned execution entity can combine the above-mentioned historical gaze point positions, the above-mentioned gaze point positions to be calibrated, and the confidences corresponding to the above-mentioned gaze point positions to be calibrated into the above-mentioned gaze point data pair. As an example: when the above-mentioned historical gaze point position is (100, 150), the above-mentioned gaze point position to be calibrated is (100, 150), and the confidence is 0.8, the obtained gaze point data pair can be ((100, 150), (100, 150), 0.8).

[0082] In the fourth sub-step, each of the obtained gaze point data pairs that satisfies the preset data condition is determined as each of the gaze point data pairs to be calibrated. The preset data condition may be that the confidence included in the gaze point data pair is greater than a preset confidence value. The confidence value may be any value within [0,1]. Here, the specific setting of the confidence value is not limited.

[0083] The fifth sub-step is to perform the following steps for each of the above-mentioned gaze point data pairs to be calibrated:

[0084] Sub-step 1: Generate a deviation result based on the historical gaze point position and the gaze point position to be calibrated included in the gaze point data pair to be calibrated. The deviation result can characterize the degree of deviation between the historical gaze point position and the gaze point position to be calibrated. In practice, the execution subject can obtain the distance between the historical gaze point position and the gaze point position to be calibrated by the distance formula between two points. Then, the obtained distance can be determined as the deviation result.

[0085] Sub-step two, in response to determining that the above-mentioned deviation result satisfies the preset deviation condition, based on the above-mentioned deviation result, the above-mentioned eye movement information generation algorithm is self-calibrated to update the above-mentioned eye movement information generation algorithm. Among them, the above-mentioned deviation condition can be that the deviation result is greater than the preset deviation value. The above-mentioned deviation value can be a pre-set value. Here, the specific setting of the above-mentioned deviation value is not limited. In practice, the above-mentioned execution subject can use the gradient descent method (such as a small batch gradient descent algorithm) to adjust the parameters of the above-mentioned eye movement information generation algorithm to perform self-calibration processing on the above-mentioned eye movement information generation algorithm.

[0086] The above technical solution and its related contents, as an inventive point of the embodiment of the present disclosure, solve the technical problem 2 mentioned in the background technology: "With the increase of the number of interactions, the reliability of the eye tracking algorithm gradually decreases, resulting in the gradual decrease in the accuracy of the eye movement data generated by the eye tracking algorithm, which in turn leads to the low reliability of the human-computer interaction results determined by the eye movement data, resulting in a high error rate of the executed human-computer interaction tasks, resulting in a waste of computing resources when executing the human-computer interaction tasks." The factors that lead to the high error rate of the executed human-computer interaction tasks and the waste of computing resources when executing the human-computer interaction tasks are often as follows: With the increase of the number of interactions, the reliability of the eye tracking algorithm gradually decreases, resulting in the gradual decrease in the accuracy of the eye movement data generated by the eye tracking algorithm, which in turn leads to the low reliability of the human-computer interaction results determined by the eye movement data, resulting in a high error rate of the executed human-computer interaction tasks, resulting in a waste of computing resources when executing the human-computer interaction tasks. If the above factors are solved, the error rate of the executed human-computer interaction tasks can be reduced, and the waste of computing resources when executing the human-computer interaction tasks can be reduced. In order to achieve this effect, the present disclosure first receives a self-calibration information group. Thus, the conditions that need to be met for self-calibration can be obtained. Then, in response to determining that the self-calibration information group meets the preset self-calibration conditions, the following steps are performed: First, historical human-computer interaction data is obtained, wherein the historical human-computer interaction data includes historical eye movement data and historical image processing results. Thus, the original data of self-calibration can be obtained. Then, based on the historical human-computer interaction data, each historical gaze point position is generated. Thus, each historical gaze point position can be obtained. Then, for each of the above historical gaze point positions, the following steps are performed: First, based on the historical human-computer interaction data, the gaze point position to be calibrated corresponding to the above historical gaze point position is generated, wherein the above gaze point position to be calibrated corresponds to a confidence. Thus, each gaze point position to be calibrated corresponding to each historical point position can be obtained, and the gaze point position to be calibrated can be screened by the confidence in subsequent processing. Secondly, the above historical gaze point position, the above gaze point position to be calibrated and the confidence corresponding to the above gaze point position to be calibrated are integrated and processed to obtain a gaze point data pair. Thus, a gaze point data pair can be obtained. Then, each of the obtained gaze point data pairs that meet the preset data conditions is determined as each of the gaze point data pairs to be calibrated. Thus, the gaze point data pairs can be screened, and the gaze point data pairs with higher confidence can be screened out as data for self-calibration processing of the algorithm. Then, for each of the above-mentioned gaze point data pairs to be calibrated, the following steps are performed: Then, based on the historical gaze point positions and the gaze point positions to be calibrated included in the above-mentioned gaze point data pairs to be calibrated, a deviation result is generated. Thus, the deviation between the generated gaze point position and the real gaze point position can be obtained.Finally, in response to determining that the above deviation result satisfies the preset deviation condition, the above eye movement information generation algorithm is self-calibrated based on the above deviation result to update the above eye movement information generation algorithm. Thus, the parameters of the eye movement information generation algorithm for generating the gaze point position can be adjusted according to the deviation result. Also, because, when the preset self-calibration condition is met, the above execution subject can self-calibrate the eye movement information generation algorithm according to the deviation result between the generated gaze point position and the actual gaze point position, thus, the reliability of the algorithm can be improved, the accuracy of the eye movement data generated by the algorithm can be improved, and then the reliability of the human-computer interaction results determined by the eye movement data can be improved, the error rate of the executed human-computer interaction tasks can be reduced, and the waste of computing resources when executing human-computer interaction tasks can be reduced.

[0087] In some optional implementations of some embodiments, the execution subject may perform the human-computer interaction task corresponding to the first human-computer interaction information based on the first human-computer interaction information through the following steps:

[0088] The first step is to send the first human-computer interaction information to a user terminal connected in communication.

[0089] In the second step, in response to receiving the feedback information sent by the user terminal, the following steps are performed:

[0090] The first sub-step is to perform voice conversion processing on the feedback information to obtain feedback voice data. The feedback information may be a label used to characterize whether the execution process is smooth. For example, the feedback information may be "operation successful" or "operation failed". The feedback voice data may be audio data corresponding to the feedback information. In practice, the execution subject may convert the feedback information into corresponding feedback voice data through TTS technology.

[0091] The second sub-step is to play the above-mentioned feedback voice data.

[0092] Step 1053, in response to detecting voice information, performing the following steps:

[0093] Step 10531, performing data conversion processing on the voice information to obtain voice content information corresponding to the voice information.

[0094] In some embodiments, the execution subject may perform data conversion processing on the voice information to obtain voice content information corresponding to the voice information. The voice content information may be text corresponding to the voice information. In practice, the execution subject may convert the voice information into corresponding voice content information through automatic voice recognition technology.

[0095] Step 10532, based on the image processing result, the voice content information, the call information group and the prompt information group, generate corresponding second human-computer interaction information.

[0096] In some embodiments, the execution subject may generate corresponding second human-computer interaction information based on the image processing result, the voice content information, the call information group and the prompt information group. The method of generating the second human-computer interaction information may refer to the specific implementation method of step 1051, which will not be repeated here.

[0097] Step 10533: Based on the second human-computer interaction information, execute the human-computer interaction task corresponding to the second human-computer interaction information.

[0098] In some embodiments, the execution subject may execute the human-computer interaction task corresponding to the second human-computer interaction information based on the second human-computer interaction information. The manner of executing the human-computer interaction task corresponding to the second human-computer interaction information may refer to the specific implementation manner of step 1052, which will not be described in detail here.

[0099] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the human-computer interaction method based on smart glasses in some embodiments of the present disclosure, the waste of computing resources during image processing can be reduced, and the processing speed during human-computer interaction can be improved. Specifically, the reason for the waste of computing resources during image processing and the slow processing speed during human-computer interaction is that when recognizing an image, it is necessary to first recognize all objects in the image, and then feed back the recognition results corresponding to the user's needs in each recognition result to the user, resulting in too many useless recognition results, which in turn leads to waste of computing resources during image processing and poor processing speed during human-computer interaction. Based on this, the human-computer interaction method based on smart glasses in some embodiments of the present disclosure first obtains scene visual data and eye movement data. Thus, the original data of human-computer interaction can be obtained. Secondly, based on the above eye movement data, eye movement information is generated, wherein the above eye movement information includes eye movement trajectory information, gaze point position and eye movement pattern information. Thus, the user's eye movement information can be obtained through the user's eye movement data. Then, based on the above eye movement information, the above scene visual data and each previously acquired historical image area to be processed, the image area to be processed is generated. Thus, the image area that the user is looking at can be determined through the user's eye movement information, and only the determined image area can be identified in the subsequent processing, which can reduce useless identification results, improve the processing speed of image processing, and further reduce the waste of computing resources during image processing, and improve the processing speed of human-computer interaction. Then, based on the above-mentioned image area to be processed, the corresponding image processing result is generated. Thus, the information corresponding to the image area that the user is looking at can be obtained. Then, in response to determining that the above-mentioned image processing result meets the preset result condition, the following steps are performed: first, based on the above-mentioned image processing result, the preset call information group and the preset prompt information group, the corresponding first human-computer interaction information is generated. Thus, the human-computer interaction information corresponding to the user's needs can be quickly generated through the above-mentioned image processing result, the above-mentioned call information group and the above-mentioned prompt information group. Then, based on the above-mentioned first human-computer interaction information, the human-computer interaction task corresponding to the above-mentioned first human-computer interaction information is executed. Thus, the function corresponding to the user's needs can be quickly called through the first human-computer interaction information, the human-computer interaction task corresponding to the user's needs can be quickly executed, and the processing speed during human-computer interaction can be greatly improved. Then, in response to detecting the voice information, the following steps are performed: first, the voice information is subjected to data conversion processing to obtain the voice content information corresponding to the voice information. Thus, the user's voice information can be converted into text information. Then, based on the image processing result, the voice content information, the call information group and the prompt information group, the corresponding second human-computer interaction information is generated. Thus, the human-computer interaction information corresponding to the user's needs can be obtained. Finally, based on the second human-computer interaction information, the human-computer interaction task corresponding to the second human-computer interaction information is executed.Therefore, the human-computer interaction tasks corresponding to the user's needs can be quickly executed through the human-computer interaction information, which can greatly improve the processing speed of human-computer interaction. Also, because the image area that needs to be identified can be determined based on the user's eye movement information, and then the image area is identified, without the need to identify all image areas, it is possible to reduce useless recognition results and reduce the waste of computing resources during image processing, thereby improving the processing speed of human-computer interaction. Also, because the human-computer interaction information corresponding to the user's needs can be generated by combining the callable information group and the prompt information group, and then the human-computer interaction tasks are executed according to the human-computer interaction information, it is possible to directly determine the operation command to be executed through the generated human-computer interaction information, which can improve the processing speed of human-computer interaction.

[0100] Reference below Figure 2 , which shows a schematic structural diagram of smart glasses 200 suitable for implementing some embodiments of the present disclosure. Figure 2 The smart glasses shown are merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0101] like Figure 2 As shown, the smart glasses 200 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage device 208 to a random access memory (RAM) 203. In the RAM 203, various programs and data required for the operation of the smart glasses 200 are also stored. The processing device 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0102] Typically, the following devices may be connected to the I / O interface 205: an input device 206 including, for example, a first visual sensor, a second visual sensor, a sound collector, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 207 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; and a communication device 209. The communication device 209 may allow the smart glasses 200 to communicate with other devices wirelessly or by wire to exchange data. Figure 2 The smart glasses 200 with various devices are shown, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead. Figure 2 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0103] The smart glasses 200 may include two second visual sensors, one of which is configured to obtain left-eye eye movement data; and the other second visual sensor is configured to obtain right-eye eye movement data.

[0104] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 209, or installed from the storage device 208, or installed from the ROM 202. When the computer program is executed by the processing device 201, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0105] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0106] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0107] The computer-readable medium may be included in the smart glasses, or it may exist independently without being assembled into the smart glasses. The computer-readable medium carries one or more programs. When the one or more programs are executed by the smart glasses, the smart glasses: obtain scene visual data and eye movement data; generate eye movement information based on the eye movement data, wherein the eye movement information includes eye movement trajectory information, gaze point position and eye movement pattern information; generate an image area to be processed based on the eye movement information, the scene visual data and each previously acquired historical image area to be processed; generate a corresponding image processing result based on the image area to be processed; in response to determining that the image processing result meets a preset result condition, perform the following steps: based on the above The method comprises the steps of: generating corresponding first human-computer interaction information based on the image processing result, the preset calling information group and the preset prompt information group; executing the human-computer interaction task corresponding to the first human-computer interaction information based on the first human-computer interaction information; and executing the following steps in response to detecting voice information: performing data conversion processing on the voice information to obtain voice content information corresponding to the voice information; generating corresponding second human-computer interaction information based on the image processing result, the voice content information, the calling information group and the prompt information group; and executing the human-computer interaction task corresponding to the second human-computer interaction information based on the second human-computer interaction information.

[0108] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0109] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0110] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0111] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A human-computer interaction method based on smart glasses, comprising: Acquire scene visual data and eye movement data; Based on the eye movement data, generating eye movement information, wherein the eye movement information includes eye movement trajectory information, gaze point position and eye movement pattern information; Generate an image region to be processed based on the eye movement information, the scene visual data and each previously acquired historical image region to be processed; Based on the image area to be processed, generating a corresponding image processing result; Generate eye movement operation information based on the eye movement trajectory information and the eye movement pattern information included in the eye movement information; Performing recognition processing on the eye movement operation information to obtain an eye movement operation recognition result corresponding to the eye movement operation information; In response to determining that the eye movement operation recognition result meets the preset operation condition, performing the following steps: Based on a preset callable information mapping relationship, determining the callable information corresponding to the eye movement operation recognition result; Based on the callable information, a corresponding human-computer interaction task is executed to obtain a corresponding execution result; Performing voice conversion processing on the execution result to obtain execution result voice data corresponding to the execution result; Playing the execution result voice data; In response to determining that the image processing result meets a preset result condition, performing the following steps: Generate corresponding first human-computer interaction information based on the image processing result, a preset calling information group and a preset prompt information group, wherein the calling information in the calling information group is information corresponding to a preset custom function, and the prompt information in the prompt information group is text information for guiding output results; Based on the first human-computer interaction information, executing the human-computer interaction task corresponding to the first human-computer interaction information; In response to detecting voice information, the following steps are performed: Performing data conversion processing on the voice information to obtain voice content information corresponding to the voice information; generating corresponding second human-computer interaction information based on the image processing result, the voice content information, the call information group and the prompt information group; Based on the second human-computer interaction information, a human-computer interaction task corresponding to the second human-computer interaction information is executed.

2. The method according to claim 1, wherein: The eye movement data includes left eye movement data and right eye movement data; And generating eye movement information based on the eye movement data includes: Preprocessing the left eye movement data and the right eye movement data included in the eye movement data to obtain preprocessed eye movement data, wherein the preprocessed eye movement data includes each preprocessed left eye pupil position and each preprocessed right eye pupil position; Based on each preprocessed left eye pupil position and each preprocessed right eye pupil position included in the preprocessed eye movement data, generating each gaze point coordinate, wherein each of the gaze point coordinates corresponds to a generation time and an end time; Constructing eye movement trajectory information based on the respective gaze point coordinates and the respective generation times corresponding to the respective gaze point coordinates; For each of the gaze point coordinates, generating a gaze duration corresponding to the gaze point coordinate based on a generation time and an end time corresponding to the gaze point coordinate; For every two gaze point coordinates that meet the preset generation time condition among the gaze point coordinates, the following steps are performed: Based on the two fixation point coordinates, generating corresponding saccade amplitudes; Based on the two gaze point coordinates, generating corresponding gaze direction information; Integrate and process the generated information of each fixation duration, each saccade amplitude and each fixation direction to obtain eye movement pattern information; Determine the gaze point coordinates that meet the preset gaze condition among the gaze point coordinates as the gaze point position; The gaze point position, the eye movement trajectory information and the eye movement pattern information are integrated and processed to obtain eye movement information.

3. The method according to claim 1, wherein: The step of generating the image region to be processed based on the eye movement information, the scene visual data and each previously acquired historical image region to be processed comprises: In response to determining that the gaze point position included in the eye movement information satisfies a preset position condition, determining a historical image region to be processed that satisfies the preset region condition among the various historical image regions to be processed as an image region to be processed; In response to determining that the gaze point position included in the eye movement information does not satisfy the preset position condition, based on the gaze point position and the scene visual data, an image area of ​​a preset shape is determined as the image area to be processed, wherein the size of the image area to be processed is a preset size.

4. The method according to claim 1, wherein: The generating corresponding first human-computer interaction information based on the image processing result, the preset calling information group and the preset prompt information group includes: Based on the image processing result, the preset call information group and the preset prompt information group, generating corresponding prompt result information; In response to determining that the prompt result information meets a preset information condition, determining the prompt result information as first human-computer interaction information; In response to determining that the prompt result information does not meet the preset information condition, the prompt result information is cleared and corresponding clearing prompt information is generated.

5. The method according to claim 1, wherein: The performing the human-computer interaction task corresponding to the first human-computer interaction information based on the first human-computer interaction information includes: Sending the first human-computer interaction information to a user terminal connected in communication; In response to receiving the feedback information sent by the user terminal, performing the following steps: Performing voice conversion processing on the feedback information to obtain feedback voice data; The feedback voice data is played.

6. A smart glasses comprising: A first visual sensor configured to acquire scene visual data; a second visual sensor configured to acquire eye movement data; A sound collector is configured to detect voice information; one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

7. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images