Human-computer interaction method and intelligent glasses

Abstract

The invention provides a man-machine interaction method and intelligent glasses, which not only can capture interaction information of a user in real time and realize interaction with the user based on the interaction information, but also can recommend an interaction mode matched with the complexity of the content of the current display interface of the intelligent glasses to the user. The man-machine interaction method comprises the following steps: performing complexity evaluation on display content and / or a current application scene of a current display interface of the intelligent glasses to obtain a complexity level of the current display content; and determining an interaction mode corresponding to the complexity level of the current display content according to a preset corresponding relationship between the complexity level and each interaction mode, and recommending the interaction mode to a user.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a human-computer interaction method and smart glasses. Background Art

[0002] With the rapid development of speech recognition, natural language processing and artificial intelligence technologies, smart glasses have gradually become capable of supporting multiple interaction modes. By integrating advanced sensors such as high-definition cameras, depth sensors, GPS, and inertial measurement units (IMUs), smart glasses can capture user interaction information in real time and interact with users based on the interaction information. The existing interaction methods of smart glasses devices are difficult for users who have just experienced smart glasses to master. Summary of the invention

[0003] One advantage of the present application is that it provides a human-computer interaction method and smart glasses, which can recommend to users an interaction mode that matches the complexity of the content of the current display interface of the smart glasses, thereby improving the efficiency of the user's proficient use of the smart glasses.

[0004] Based on this, in order to achieve at least one of the above advantages or other advantages and purposes of the present application, the present application provides a human-computer interaction method, which is applied to smart glasses. The smart glasses support multiple interaction modes. The method includes: Performing a complexity evaluation on current display content of the display interface of the smart glasses to obtain a complexity level of the current display content; According to the correspondence between the preset complexity level and each interaction mode, the interaction mode corresponding to the complexity level of the currently displayed content is determined, and the interaction mode is recommended to the user.

[0005] In this way, the human-computer interaction method automatically determines and recommends the most suitable interaction mode to the user based on the complexity of the interface and / or the current application scenario, such as buttons or knobs, voice or gesture control, thereby reducing the user's cognitive load when understanding and operating complex interfaces, thereby improving operational efficiency and user satisfaction.

[0006] According to an embodiment of the present application, a complexity evaluation is performed on the display content of the current display interface of the smart glasses to obtain the complexity level of the current display content, including: Determining at least one application interface in the currently displayed content; Determining functional components and display information corresponding to the application interface, wherein the functional components include hidden functional components; The complexity level of the currently displayed content is determined according to the functional components and display information of the application interface.

[0007] With such a configuration, the complexity level of the currently displayed content can be determined based on the determined functional components and display information of the application interface.

[0008] According to an embodiment of the present application, determining the complexity level of the currently displayed content according to the functional components and display information of the application function includes: Determine the type and quantity of the functional components, the type of the functional components including at least one of the following: a text box, a function button, and a drop-down menu; The complexity level of the currently displayed content is determined according to the type and quantity of the functional components and the display information.

[0009] According to one embodiment of the present application, the display information includes the visual hierarchy and layout complexity of the application interface, the page operation complexity, the page operation accuracy, the page information density and the number of dynamic elements on the page, and the multiple interaction methods include ray interaction, voice interaction and gesture interaction.

[0010] With such a setting, the complexity level of the currently displayed content can be determined based on the functional components in different display interfaces, the visual hierarchy and layout complexity of the application interface, the complexity of page operations, the accuracy of page operations, the page information density, and the number of dynamic elements on the page. Furthermore, an interaction method that matches the complexity level of the currently displayed content can be determined and recommended to the user. This can reduce the cognitive load of the user when understanding and operating complex interfaces, thereby improving operation efficiency and user satisfaction.

[0011] According to an embodiment of the present application, according to the correspondence between the preset complexity level and each interaction mode, an interaction mode corresponding to the complexity level of the currently displayed content is determined, and the interaction mode is recommended to the user, specifically: According to the correspondence between the intervals divided by the preset multiple thresholds and the various interaction modes, an interaction mode corresponding to the complexity level of the currently displayed content is determined, and the interaction mode is recommended to the user.

[0012] With such settings, the complexity of the content of the current display interface of the smart glasses can be quantified by using multiple preset thresholds and the intervals divided by the multiple thresholds. By quantifying the complexity of the interface, the elements and operation steps of the display interface can be accurately evaluated, so as to more accurately determine which interfaces require more intuitive and simplified interaction methods. This can significantly improve the efficiency and accuracy of the operation and reduce the user's operation time and error rate.

[0013] According to one embodiment of the present application, the correspondence between the intervals divided by the preset multiple thresholds and the various interaction modes is that the number of the multiple thresholds is m, the number of the multiple interaction modes is n, m=2n-2, and the interval intervals matching each interaction mode are set between the multiple thresholds.

[0014] In this way, according to the number of interaction modes that the smart glasses can support, multiple thresholds are set that match the number of interaction modes that the smart glasses can support, so that the human-computer interaction method can be widely adapted to a variety of smart glasses that support different numbers of interaction modes.

[0015] According to one embodiment of the present application, after determining the interaction method corresponding to the complexity level of the currently displayed content based on the correspondence between the preset complexity level and each interaction method, and recommending the interaction method to the user, it also includes predicting the user's preferred interaction mode based on the user's habits.

[0016] With such a configuration, the human-computer interaction method can predict the user's preferred interaction method for display interfaces of different complexity levels based on the interaction data of the user since using the smart glasses. Compared with directly recommending interaction methods to the user based on the correspondence between complexity levels and various interaction methods and the complexity level of the current display interface, this solution can recommend to the user a more personalized interaction method that matches the interaction data based on the user's usage habits.

[0017] According to an embodiment of the present application, predicting the user's preferred interaction mode based on the user's habits is specifically: Save all interaction data of the user since using the smart glasses; Based on the analysis of all interaction data of the user since using the smart glasses, the interaction preference data of the user is obtained and stored; Based on the interaction preference data, a deep learning model is used to predict the user's preferred interaction methods for display interfaces of different complexity levels.

[0018] With this configuration, the interaction preference data unique to the user can be obtained and stored based on all the interaction data since the user used the smart glasses, and the interaction method preferred by the user for display interfaces of different complexity levels can be predicted based on the interaction preference data and the deep learning model. Each user has his or her own unique operating preferences and workflows. The human-computer interaction method captures these habits and accurately reflects and supports these preferences in the form of recommended interaction methods, thereby providing users with a smoother and more efficient operating experience.

[0019] According to an embodiment of the present application, after predicting the user's preferred interaction mode based on the user's habits, the method further includes: When the complexity level of the currently displayed content is between two intervals that match the interaction mode, an interaction mode is recommended to the user based on the predicted preferred interaction mode, and when the interaction mode used by the user is inconsistent with the recommended interaction mode, the user's interaction preference data is updated.

[0020] In this way, when the complexity level of the currently displayed content is between two intervals that match the interaction mode, that is, the complexity level of the currently displayed content is not within the interval that matches the interaction method, the human-computer interaction method will recommend an interaction mode to the user based on the predicted preferred interaction mode, which can significantly improve the user experience and operation efficiency. By deeply analyzing the user's specific usage habits, the human-computer interaction method can provide a highly personalized interaction mode that accurately meets the user's needs. And when the interaction mode used by the user is inconsistent with the recommended interaction mode, the user's interaction preference data is updated, so that when the interaction mode is recommended to the user next time, an interaction mode that is more in line with the user's habits can be recommended to the user, thereby improving the efficiency of the operation and the user's satisfaction.

[0021] According to another aspect of the present application, the present application further provides a pair of smart glasses, comprising: a sensor, a display module, a memory and a processor, the sensor being used to receive interaction information sent by a user to the smart glasses, the display module being used to display an interface to the user, the memory storing a computer program, and the computer program realizing the functions of the above-mentioned human-computer interaction method when executed by the processor.

[0022] With such configuration, the sensor, display module, memory and processor cooperate together to implement the functions of the above-mentioned human-computer interaction method.

[0023] Beneficial effect: the human-computer interaction method can automatically determine the most suitable interaction mode, such as buttons or knobs, voice or gesture control, according to the complexity of the interface and / or the current application scenario, to reduce the cognitive load of users in understanding and operating complex interfaces, thereby improving operational efficiency and user satisfaction. Secondly, the human-computer interaction method can significantly enhance operational efficiency, especially for interfaces that are information-intensive or require multi-step operations, by recommending more intuitive and efficient operating methods. In addition, the human-computer interaction method can recommend users to use an interaction method that matches the current display interface, thereby improving the overall use efficiency of the system. Another advantage is that this intelligent recommendation can provide personalized adaptation. According to the habits and preferences of different users, the system will recommend the interaction method that best meets their personal needs, so that each user can get a personalized operating experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A flowchart of a human-computer interaction method provided for one embodiment of the present application; Figure 2 An application environment diagram of a human-computer interaction method provided by an embodiment of the present application; Figure 3 A schematic flow chart of step S1 in a human-computer interaction method provided in another embodiment of the present application; Figure 4 A schematic flow chart of step S13 in a human-computer interaction method provided in another embodiment of the present application; Figure 5 A schematic flow chart of step S3 in a human-computer interaction method provided in another embodiment of the present application; Figure 6 A module block diagram of smart glasses provided for another embodiment of the present application; Figure 7 An internal structure diagram of a computer device provided for another embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0027] Considering that although existing smart glasses can interact with users, the interaction methods they support are often relatively simple, and it is difficult to adapt to the multi-tasking requirements in complex application scenarios. In addition, for users who have just experienced smart glasses, there is a lack of guidance and it is difficult to use them proficiently. In order to solve this problem, the present application provides a human-computer interaction method, a storage medium, and smart glasses, which can not only capture the user's interaction information in real time and interact with the user based on the interaction information, but also recommend to the user an interaction mode that matches the complexity of the content of the current display interface of the smart glasses, thereby improving the user's efficiency in using the smart glasses proficiently.

[0028] For details, please refer to the attached Figure 1 An embodiment of the present application provides a human-computer interaction method, which is applied to smart glasses. The smart glasses support multiple interaction modes. The method includes the following steps: Step S1: performing a complexity evaluation on the current display content of the display interface of the smart glasses to obtain a complexity level of the current display content; Step S2: Determine an interaction mode corresponding to the complexity level of the currently displayed content according to the correspondence between the preset complexity level and each interaction mode, and recommend the interaction mode to the user.

[0029] In particular, the human-computer interaction method recommends interaction methods to the user in the form of recommendations including but not limited to prompts such as vibration, cursor flashing, and voice prompts.

[0030] It should be noted that the human-computer interaction method can greatly improve the user experience, because it will automatically determine the most suitable interaction mode, such as buttons or knobs, voice or gesture control, based on the complexity of the interface and / or the current application scenario, to reduce the cognitive load of users in understanding and operating complex interfaces, thereby improving operational efficiency and user satisfaction. Secondly, the human-computer interaction method can significantly enhance operational efficiency, especially for interfaces that are information-intensive or require multi-step operations. By recommending more intuitive and efficient operating methods, users can complete tasks more quickly. In addition, the human-computer interaction method can recommend users to use an interaction method that matches the current display interface, thereby improving the overall use efficiency and user stickiness of the system. Another advantage is that this intelligent recommendation can provide personalized adaptation. According to the habits and preferences of different users, the system will recommend the interaction method that best meets their personal needs, so that each user can get a personalized operating experience.

[0031] In particular, the human-computer interaction method changes based on the complexity of the page display, and performs corresponding guidance processing through logical judgment. For example, the human-computer interaction method involving system-level basic service functions and hardware sensor functions will be carried out in the form of voice commands, and flashing prompts will be given on the user's display interface. The basic service functions include but are not limited to power on, power off, settings, weather, calendar, time, and to-do, etc., and calling sensor-related functions includes but is not limited to taking pictures, recording, etc. Moreover, in scenarios where human-computer interactions involving specific functions use voice commands and gesture operations to participate together, the human-computer interaction method will give corresponding prompts according to the actual interaction process. Specific functions include: office, video editing, shopping consumption, education and learning, and film and television entertainment.

[0032] The human-computer interaction method provided in the embodiment of the present application can be applied to Figure 2 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. According to the complexity of the displayed interface and / or the current application scenario, the most suitable interaction mode, such as ray, voice or gesture control, can be automatically predicted to reduce the cognitive load of the user when understanding and operating the complex interface, thereby improving the operation efficiency and user satisfaction. Secondly, the human-computer interaction method can significantly enhance the operation efficiency, especially for information-intensive or multi-step operation interfaces, by recommending more intuitive and efficient operation methods, users can complete tasks more quickly. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0033] In other embodiments of the present application, Figure 3 As shown, a complexity evaluation is performed on the display content of the current display interface of the smart glasses to obtain the complexity level of the current display content, including: Step S11: determining at least one application interface in the currently displayed content; It is worth noting that a person skilled in the art should know that the displayed interface is the display interface of the smart glasses or an application interface in the display interface of the smart glasses, and in the application the page and the interface are equivalent.

[0034] It should be noted that compared with the interface displayed by traditional display devices, the field of view of the interface displayed by the smart glasses is larger, so there will be multiple application interfaces coexisting, and the user can select the target application interface the user wants to operate through the interaction method supported by the smart glasses.

[0035] Step S12: determining the functional components and display information corresponding to the application interface, wherein the functional components include hidden functional components; Step S13: Determine the complexity level of the currently displayed content according to the functional components and display information of the application interface.

[0036] Exemplarily, the multiple interaction modes supported by the smart glasses include ray interaction. The user can use a controller that can emit infrared rays or other controllers that can establish communication with the smart glasses (such as a mobile phone) to select the target application interface that the user wants to operate with the help of the buttons set on the controller. For example, the controller is provided with an interface selection shortcut key, four direction buttons and a confirmation button. When the user presses the interface selection shortcut key, the smart glasses use the application interface as an operable element, and the border of the interface that the user last operated will be bold and flashing to prompt the user that the interface is the current target application interface. The user can use the four direction buttons to select the application interface that he wants to operate and use it as the target application interface. With the help of the confirmation button, the application interface is selected, and the smart glasses use the functional components or other elements in the application interface as operable elements. It should be noted that the above-mentioned buttons can be physical buttons or virtual buttons.

[0037] The smart glasses support a variety of interaction methods including voice interaction. The user can use the voice command rules and templates set in the smart glasses to select the target application interface that the user wants to operate. For example, the user can wake up the AI ​​assistant integrated in the smart glasses through a preset wake-up word, and enter the target interface selection mode with the help of the keyword "select interface", that is, the smart glasses mentioned above use the application interface as an operable element, and the border of the interface that the user last operated will be bolded and flashing to prompt the user that the interface is the current target application interface. Then the user can use relevant directional words (such as up, down, left, right, front and back) to select the application interface that the user wants to operate as the target application interface, and then use the keyword "OK" to select the application interface. The smart glasses use the functional components or other elements in the application interface as operable elements.

[0038] The multiple interaction modes supported by the smart glasses include gesture interaction. The user can use the gesture recognition rules and templates set in the smart glasses to select the target application interface that the user wants to operate. For example, the user can select the application interface that the user wants to operate through a preset target interface selection gesture, such as a grasping gesture, and use it as the target application interface. The smart glasses use the functional components or other elements in the application interface as operable elements.

[0039] The smart glasses support multiple interaction modes including eye movement interaction. The smart glasses capture images of the eyes and identify features such as the eyeball, pupil and corneal reflection, and use the interface the user is looking at as the target application interface.

[0040] In other embodiments of the present application, Figure 4 As shown, determining the complexity level of the currently displayed content according to the functional components of the application function and the display information includes: Step S131: Determine the type and quantity of the functional components, the type of the functional components at least includes one of the following: text box, function button, drop-down menu; Step S132: Determine the complexity level of the currently displayed content according to the type and quantity of the functional components and the display information.

[0041] In other embodiments of the present application, the display information includes the visual level and layout complexity of the application interface, the page operation complexity, the page operation accuracy, the page information density, and the number of page dynamic elements, and the multiple interaction methods include ray interaction, voice interaction, and gesture interaction. Based on the functional components in different display interfaces, the visual level and layout complexity of the application interface, the page operation complexity, the page operation accuracy, the page information density, and the number of page dynamic elements, the complexity level of the current display content can be determined, and then the interaction method that matches the complexity level of the current display content can be determined, and the interaction method can be recommended to the user, which can reduce the cognitive load of the user when understanding and operating a complex interface, so as to improve the efficiency of the operation and the user's satisfaction.

[0042] Specifically, the complexity level of the displayed interface is closely related to the number and type of functional components displayed on the page. The smart glasses will tend to increase or decrease the complexity level of the displayed interface according to the number, type and interactive depth of each functional component in the displayed interface. Exemplarily, the smart glasses will automatically identify the type and number of each functional component on the displayed interface, and calculate the complexity score of the displayed interface in terms of functional components based on the preset complexity weights of each type of functional component. The complexity weights of each type of functional component are pre-set and are set based on the interactive depth and / or operable level, type and area size of each type of functional component. For example, for a function button and a drop-down menu of equal area, because the interactive depth and / or operable level of the drop-down menu is higher, the complexity weight of the drop-down menu of equal volume is higher. In a page including only one function button and a page including only one drop-down menu, the complexity score of the page including only one drop-down menu in terms of functional components is relatively higher. The functional components include but are not limited to: text boxes, function buttons, drop-down menus, check boxes, sliders, switches, draggable items, modal boxes, progress bars, scroll bars, date selectors, and color selectors, etc.

[0043] Visual hierarchy and layout complexity: that is, the layout complexity of the page, such as hierarchical display effects, nested elements or scrolling content, etc., will also affect the results of the complexity assessment; illustratively, the smart glasses will perform a complexity assessment of the visual hierarchy and layout complexity of the displayed interface according to the visual hierarchy and layout complexity in the displayed interface, and obtain a complexity score of the displayed interface in terms of the visual hierarchy and layout complexity. For example, a page with fewer colors and no page segmentation and a page with more colors (colorful) and page segmentation, the page with more colors (colorful) and page segmentation has a higher complexity score in terms of visual hierarchy and layout complexity.

[0044] Page operation complexity: For interfaces that require sophisticated operations, such as pages that require users to perform complex operations such as dragging, moving objects, and adjusting values ​​on the screen, the page will have a higher complexity score in terms of page operation complexity compared to interfaces that do not require sophisticated operations. The smart glasses will perform a complexity assessment on the operations supported by the displayed interface based on the content in the displayed interface, thereby obtaining a complexity score for the displayed interface in terms of page operation complexity; for example, an interface that only supports button click operations and an interface that supports button clicks, slider sliding, and text searches; the interface that supports button clicks, slider sliding, and text searches will have a higher complexity score in terms of page operation complexity.

[0045] Page operation accuracy: If the displayed interface supports operations with higher accuracy (for example, clicking small buttons or adjusting small parameters), its complexity score in terms of page operation accuracy will increase accordingly. Or if the displayed interface supports operations with higher input accuracy (such as virtual keyboards, fine-tuning parameters, etc.), its complexity score in terms of page operation accuracy will increase accordingly. If the interface only supports simple operations such as confirmation and cancellation, its complexity score in terms of page operation accuracy will decrease accordingly.

[0046] Page information density: that is, the amount of content displayed on the displayed interface, the amount of text, the density of icons, the degree of icon refinement, and the ratio of image elements to the field of view size of the displayed interface. The inclusion of a large amount of page information in the displayed interface means that the user needs to spend more attention to understand the information contained in the interface, and the complexity score of the interface in terms of page information density will also increase accordingly. For example, the smart glasses will calculate the page information contained in the displayed interface and the display field of view size of the displayed interface based on the preset page information density evaluation rules, thereby obtaining the complexity score of the displayed interface in terms of page information density. The page information density evaluation rules are preset, and corresponding information weights are set for the number of displayed contents, the amount of text, the density of icons, the degree of icon refinement, and the image elements in the interface. For example, every 10 words are recorded as 1 information weight, and each displayed content is recorded as 1 information weight. In addition, a field of view weight is set for the field of view size of the displayed interface. For example, every A4-sized interface is recorded as 1 field of view weight. The total information weight of the displayed interface is calculated and divided by the field of view weight of the displayed page to obtain the complexity score of the displayed page in terms of page information density.

[0047] Number of dynamic elements on the page: If the displayed page contains a large number of visual effects such as animation, dynamic feedback, etc., its complexity score in terms of the number of dynamic elements on the page will increase accordingly. The smart glasses will perform a complexity assessment based on the number of dynamic elements included in the displayed page, the playback speed, and the playback duration, thereby obtaining a complexity score for the displayed page in terms of the number of dynamic elements on the page. The logic is that the more dynamic elements are included, the faster the dynamic elements are played, and the longer the dynamic elements are played, the complexity score of the displayed interface in terms of the number of dynamic elements on the page will increase accordingly.

[0048] It is worth noting that the user will determine the complexity level of the displayed interface based on the preset complexity evaluation weights of various aspects and the complexity scores of the displayed interface in various aspects. For example, the preset complexity evaluation weights of various aspects can be divided in equal proportion, that is, the complexity evaluation weights of various aspects are all 1 / 6, and the complexity evaluation weight of a certain aspect can also be increased / decreased in a trend according to the user's preference.

[0049] In other embodiments of the present application, the complexity level of the displayed interface is a complexity score of the displayed interface in a certain aspect.

[0050] With the help of preset voice command rules and templates, users can quickly operate system-level functions (such as power on, power off, settings, calendar, weather, etc.) and hardware control (such as taking photos, etc.) through voice interaction, simplifying the execution process of common tasks. In addition, the mechanism of integrating multiple interaction methods, in scenarios where the complexity level of the displayed interface is high, multiple interaction methods are combined to interact with the displayed interface. The system automatically adjusts prompts and interaction methods according to the complexity of the task to ensure that the user's instructions can be accurately responded to. Dynamic interaction recommendation mechanism, the system will introduce page complexity judgment logic, judge the complexity level of the page based on multiple preset thresholds, make corresponding recommendations for interaction methods based on the complexity level, and provide multi-modal interaction, which can adapt to different task complexities. In addition, it can also recommend more efficient interaction methods for users based on the recommendation system based on user habits, so as to reduce the user's adaptation time and improve the user's convenience in using smart glasses.

[0051] It is worth noting that the multiple interaction modes include but are not limited to ray interaction, voice interaction, gesture interaction, eye movement interaction, etc. In this embodiment, multiple interaction modes including ray interaction, voice interaction, and gesture interaction are used as examples for illustration.

[0052] In particular, the voice interaction is based on natural language processing technology, which responds to the user's voice commands or questions by setting voice command rules and templates; the gesture interaction is based on the sensors and computer vision technology in the smart glasses, which responds to the user's gesture commands through real-time image processing.

[0053] In other embodiments of the present application, according to the correspondence between the preset complexity level and each interaction mode, an interaction mode corresponding to the complexity level of the currently displayed content is determined, and the interaction mode is recommended to the user, specifically: Step S21: Determine an interaction mode corresponding to the complexity level of the currently displayed content according to the correspondence between the intervals divided by the preset multiple thresholds and the various interaction modes, and recommend the interaction mode to the user.

[0054] It should be noted that the complexity of the content of the current display interface of the smart glasses can be quantified by using multiple preset thresholds and the intervals divided by the multiple thresholds. By quantifying the complexity of the interface, the elements and operation steps of the display interface can be accurately evaluated, so as to more accurately determine which interfaces require more intuitive and simplified interaction methods. This can significantly improve the efficiency and accuracy of the operation and reduce the user's operation time and error rate.

[0055] In other embodiments of the present application, the correspondence between the intervals divided by the preset multiple thresholds and the various interaction modes is that the number of the multiple thresholds is m, the number of the multiple interaction modes is n, m=2n-2, and the intervals matching the various interaction modes are set between the multiple thresholds. According to the number of interaction modes supported by the smart glasses, multiple thresholds matching the number of interaction modes supported by the smart glasses can be set, so that the human-computer interaction method can be widely adapted to a variety of smart glasses supporting different numbers of interaction modes.

[0056] It is worth noting that the smart glasses support n kinds of interaction modes, and there are multiple application scenarios for the smart glasses. In a certain target application scenario, k kinds of interaction modes are adapted, and k is less than or equal to n. For the same smart glasses, different application scenarios may require different numbers of interaction modes. In this case, the method can also set matching threshold numbers for multiple interaction modes adapted for different scenarios, thereby significantly improving the flexibility and applicability of the human-computer interaction method, and being able to flexibly adjust the threshold number according to specific application scenarios and user needs, ensuring that the human-computer interaction method can be accurately adapted to various complex interface situations.

[0057] It is worth noting that when the multiple interaction methods include ray interaction, voice interaction and gesture interaction, the number of thresholds is 4. According to the correspondence between the intervals divided by the preset multiple thresholds and the various interaction methods, the interaction method corresponding to the complexity level of the currently displayed content is determined, and the interaction method is recommended to the user. Specifically: when the complexity level of the currently displayed content is lower than the first threshold, the human-computer interaction method recommends the user to use ray interaction; when the complexity level of the currently displayed content is higher than the second threshold and lower than the third threshold, the human-computer interaction method recommends the user to use voice interaction; when the complexity level of the currently displayed content is higher than the fourth threshold, the human-computer interaction method recommends the user to use gesture interaction.

[0058] In other embodiments of the present application, the human-computer interaction method can determine whether the complexity level of the currently displayed content is within a range that matches the interaction method. If it is within the range, it is determined that the interaction method to be recommended is an interaction method that matches the range where the complexity level is located.

[0059] In other embodiments of the present application, Figure 5 As shown, after determining the interaction mode corresponding to the complexity level of the currently displayed content according to the correspondence between the preset complexity level and each interaction mode, and recommending the interaction mode to the user, it also includes step S3: predicting the user's preferred interaction mode based on the user's habits, specifically: Step S31: saving all interaction data of the user since using the smart glasses; Step S32: analyzing all interaction data of the user since using the smart glasses, obtaining and storing the interaction preference data of the user; All interaction data since the user used the smart glasses are saved, and all the interaction data are analyzed to obtain the interaction preference data of the user. Compared with the solution of only saving and analyzing part of the interaction data since the user used the smart glasses, this solution analyzes all the interaction data since the user used the smart glasses, and the amount of interaction data is larger, thereby preventing other users from using the smart glasses at some time to affect the predicted preferred interaction mode for display interfaces with different complexity levels, thereby improving the accuracy of the predicted preferred interaction mode.

[0060] Step S33: Based on the interaction preference data, predict the user's preferred interaction mode for display interfaces of different complexity levels through a deep learning model.

[0061] Based on all the interaction data since the user used the smart glasses, the interaction preference data unique to the user can be obtained and stored, and the interaction method preferred by the user for display interfaces of different complexity levels can be predicted based on the interaction preference data and the deep learning model. Compared with the solution that directly recommends interaction methods to users based on the correspondence between complexity levels and various interaction methods and the complexity level of the current display interface, this solution can recommend to users more personalized interaction methods that match the interaction data based on the user's usage habits. Each user has his or her own unique operating preferences and workflows. The human-computer interaction method captures these habits and accurately reflects and supports these preferences in the form of recommended interaction methods, thereby providing users with a smoother and more efficient operating experience.

[0062] In particular, the interaction data includes but is not limited to: the frequency with which users use various interaction modes, the interaction methods selected by users on interfaces of different complexity, the success rate and / or response time of interacting with users, and the user's default interaction method for a certain operation.

[0063] Specifically, the frequency with which users use various interaction modes: the smart glasses will record the frequency with which users use a certain interaction method (such as gestures, voice), and recommend corresponding interaction methods based on the user's preferences in different scenarios.

[0064] The interaction methods selected by users on interfaces of different complexity: The smart glasses will track the interaction methods selected by users on pages of different complexity, for example, whether users prefer gesture operation on high-complexity pages.

[0065] Success rate and / or response time of interaction with the user: The smart glasses will record the success rate and response time of the user's operations, determine whether incorrect operations occur frequently, and adjust the recommended interaction method accordingly.

[0066] The user's default interaction method for a certain operation: The smart glasses will record the user's preferred interaction method for specific operations (such as virtual keyboard input, frequently used system settings), especially after the user has modified the default settings multiple times.

[0067] In other embodiments of the present application, after predicting the user's preferred interaction mode based on the user's habits, the method further includes: Step S4: When the complexity level of the currently displayed content is between two intervals that match the interaction mode (i.e., the two interaction modes corresponding to the two intervals are both applicable to interacting with the current display interface), an interaction mode is recommended to the user based on the predicted preferred interaction mode, and when the interaction mode used by the user is inconsistent with the recommended interaction mode, the user's interaction preference data is updated.

[0068] It should be noted that when the interaction mode used by the user is inconsistent with the recommended interaction mode, the smart glasses will update the user's interaction preference data to increase the weight of the interaction mode used by the user when predicting the user's preferred interaction mode, thereby improving the accuracy of predicting the user's preferred interaction mode next time.

[0069] For example, when a user watches teaching videos or reads learning materials through smart glasses, the system will recommend voice or gesture operations based on the complexity of the page. When the user selects voice commands to turn pages multiple times, the system will automatically record the user's habits and prioritize the voice command interaction method in similar scenarios in the future, providing users with a more suitable personalized learning experience.

[0070] Exemplarily, predicting a user's preferred interaction mode based on the user's habits includes the following steps: Collect and analyze user interaction data, that is, by analyzing the user's operating habits and interface complexity, determine whether the interface is suitable for voice or gesture operation based on the threshold. The collected interaction data can be used to establish interaction preference data.

[0071] Predictive training is performed based on deep learning models and user interaction data, that is, based on recorded interaction data, the model is trained to learn user habits. For example, reinforcement learning or classification models are used to predict the best interaction methods in scenarios of different complexity.

[0072] Dynamically recommend to users based on the prediction results, that is, when the system detects that the complexity level of the currently displayed content is not within the range that matches the interaction method, it provides users with a suitable interaction method based on the prediction results of the reference model. If the user's preferences are relearned by the model, the system will give priority to recommending the habitual operation method in future operations.

[0073] In the case where the complexity level of the currently displayed content is between two intervals that match the interaction mode, that is, the complexity level of the currently displayed content is not within the interval that matches the interaction method, the human-computer interaction method will recommend an interaction mode to the user based on the predicted preferred interaction mode, which can significantly improve the user's experience and operation efficiency. By deeply analyzing the user's specific usage habits, the human-computer interaction method can provide a highly personalized interaction mode that accurately meets the user's needs. This personalized service can greatly enhance the user's satisfaction and acceptance of the system. And when the interaction mode used by the user is inconsistent with the recommended interaction mode, the user's interaction preference data is updated, so that when the interaction mode is recommended to the user next time, an interaction mode that is more in line with the user's habits can be recommended to the user, thereby improving the efficiency of the operation and the user's satisfaction.

[0074] Another embodiment of the present application provides a smart glasses, such as Figure 6 As shown, the smart glasses include: a sensor, a display module, a memory and a processor, the sensor is used to receive interaction information sent by the user to the smart glasses, the display module is used to display an interface to the user, and the memory stores a computer program, which realizes the functions of the above-mentioned human-computer interaction method when executed by the processor. The sensor, display module, memory and processor can work together to realize the functions of the above-mentioned human-computer interaction method.

[0075] In particular, the current application scenario of the smart glasses includes but is not limited to: scenarios with requirements for response time, etc. Exemplarily, when the current application scenario of the smart glasses is a scenario with requirements for response time, such as a game scenario, the human-computer interaction method will appropriately adjust the result of its complexity evaluation so that the interaction method matching the result of the complexity evaluation tends to be an interaction method with a shorter interaction time with the smart glasses.

[0076] In another embodiment of the present application, a computer device is also provided, and its internal structure diagram can be as follows: Figure 7As shown. The computer device includes a processor, a memory and a communication interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store interaction data, interaction preference data and other related data. The communication interface of the computer device is used to establish communication with an external terminal through a network. When the computer program is executed by the processor, a human-computer interaction method is implemented.

[0077] In another embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the functions of the above-mentioned human-computer interaction method are realized.

[0078] It should be noted that the computer program stored in the storage medium can automatically perform complex tasks, reduce human intervention, thereby improving overall efficiency and ensuring the reliability and trustworthiness of operations. In addition, the computer program is repeatable and extensible, so that the same computer program can be repeatedly executed in different environments and at different times, and is easy to expand to meet new needs, thereby enhancing the adaptability and flexibility of the computer program. In addition, running the computer program can greatly optimize the utilization of resources and reduce manpower and error costs by automatically processing a large amount of data and tasks. Finally, the computer program also has traceability and security, and can ensure the auditability and security of operations by recording and tracking each step of the computer program execution, thereby reducing the risks brought by human operations.

[0079] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0080] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0081] Unless otherwise defined, all technical and scientific terms used in the specification of this application have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in the specification of this application includes any and all combinations of one or more related listed items.

[0082] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. In addition, the above-mentioned embodiments only express several implementation methods of the present application, and the description is relatively specific and detailed, but it cannot be understood as a limitation on the scope of the patent of this application. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several deformations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be based on the attached claims.

Claims

1. A human-computer interaction method, characterized in that: Applied to smart glasses, the smart glasses support multiple interaction modes, the method includes: Performing a complexity evaluation on current display content of the display interface of the smart glasses to obtain a complexity level of the current display content; According to the correspondence between the preset complexity level and each interaction mode, the interaction mode corresponding to the complexity level of the currently displayed content is determined, and the interaction mode is recommended to the user.

2. The human-computer interaction method according to claim 1, characterized in that: A complexity evaluation is performed on the display content of the current display interface of the smart glasses to obtain the complexity level of the current display content, including: Determining at least one application interface in the currently displayed content; Determining functional components and display information corresponding to the application interface, wherein the functional components include hidden functional components; The complexity level of the currently displayed content is determined according to the functional components and display information of the application interface.

3. The human-computer interaction method according to claim 2, characterized in that: Determining the complexity level of the currently displayed content according to the functional components and display information of the application interface includes: Determine the type and quantity of the functional components, the type of the functional components including at least one of the following: a text box, a function button, and a drop-down menu; The complexity level of the current display content is determined according to the type and quantity of the functional components and the display information.

4. The human-computer interaction method according to claim 3, characterized in that: The display information includes the visual hierarchy and layout complexity of the application interface, the page operation complexity, the page operation accuracy, the page information density and the number of dynamic elements on the page. The multiple interaction methods include ray interaction, voice interaction and gesture interaction.

5. The human-computer interaction method according to claim 1, characterized in that: According to the correspondence between the preset complexity level and each interaction mode, determining the interaction mode corresponding to the complexity level of the currently displayed content, and recommending the interaction mode to the user, specifically: According to the correspondence between the intervals divided by the preset multiple thresholds and the various interaction modes, an interaction mode corresponding to the complexity level of the currently displayed content is determined, and the interaction mode is recommended to the user.

6. The human-computer interaction method according to claim 5, characterized in that: The correspondence between the intervals divided by the preset multiple thresholds and the various interaction modes is that the number of the multiple thresholds is m, the number of the multiple interaction modes is n, m=2n-2, and the intervals matching each interaction mode are set between the multiple thresholds.

7. The human-computer interaction method according to claim 6, characterized in that: The method further includes, after determining the interaction mode corresponding to the complexity level of the currently displayed content according to the correspondence between the preset complexity level and each interaction mode, and recommending the interaction mode to the user, predicting the user's preferred interaction mode based on the user's habits.

8. The human-computer interaction method according to claim 7, characterized in that: The prediction of the user's preferred interaction mode based on the user's habits is specifically as follows: Save all interaction data of the user since using the smart glasses; Based on the analysis of all interaction data of the user since using the smart glasses, the interaction preference data of the user is obtained and stored; Based on the interaction preference data, a deep learning model is used to predict the user's preferred interaction methods for display interfaces of different complexity levels.

9. The human-computer interaction method according to claim 8, characterized in that: After predicting the user's preferred interaction mode based on the user's habits, the method further includes: When the complexity level of the currently displayed content is between two intervals that match the interaction mode, an interaction mode is recommended to the user based on the predicted preferred interaction mode, and when the interaction mode used by the user is inconsistent with the recommended interaction mode, the user's interaction preference data is updated.

10. A pair of smart glasses, characterized in that: The smart glasses include: a sensor, a display module, a memory and a processor, the sensor is used to receive interaction information sent by a user to the smart glasses, the display module is used to display an interface to the user, and the memory stores a computer program, which, when executed by the processor, implements the function of the human-computer interaction method as described in any one of claims 1 to claim 9.

Images