Message processing method and device based on intelligent wearable device, equipment and medium

By utilizing eye movement data and line of sight focus position, the smart wearable device realizes notification message processing based on eye movement tracking, solving the problems of cumbersome operations and external interference in the prior art, and improving convenience and user experience.

CN119045652APending Publication Date: 2024-11-29ZHUHAI MOJIE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410719679.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-04
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

When processing notification messages, existing smart wearable devices require users to raise their hands, resulting in cumbersome operations and interfere with users' current activities, reducing user experience.

Method used

By obtaining the user's eye movement data and the focal position of the line of sight, the notification message is processed based on the position relationship of the extended screen, including displaying detailed content when the focal position of the line of sight overlaps, hiding or reducing the message when it does not overlap, and using eye tracking to achieve notification message handling.

Benefits of technology

The operation of smart wearable devices to process notification messages is simplified, operation convenience is improved, external environment interference is reduced, operation recognition rate and user experience are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119045652A_ABST
    Figure CN119045652A_ABST
Patent Text Reader

Abstract

The invention discloses a message processing method and device based on an intelligent wearable device, the intelligent wearable device and a storage medium, and the method comprises the steps: achieving the eye movement tracking of a user based on eye movement data and a sight focus position; in other words, the notification message is processed on the basis of the relative positional relationship between the user sight focus position and the extension screen for displaying the notification message or the eye movement data of the user. Therefore, the notification message processing is realized based on eye movement tracking, the operation of processing the notification message by the intelligent wearable device is simplified, the operation convenience is improved, the intelligent wearable device is not easily interfered by the external environment, the operation recognition rate is improved, and the user experience is further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of smart wearable technology, and in particular to a message processing method and apparatus based on a smart wearable device, a smart wearable device, and a computer-readable storage medium. Background Art

[0002] With the development of science and technology, augmented reality technology has been widely used in various fields. As an important carrier of this technology, smart wearable devices not only provide users with a rich visual experience, but also greatly facilitate people's daily life. However, when processing notification messages, existing smart wearable devices require users to raise their hands to operate, which is not only cumbersome to operate, but also interferes with the user's current activities and reduces the user experience. Therefore, how to improve the convenience of message processing operations of smart wearable devices has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The present application provides a message processing method and apparatus based on a smart wearable device, a smart wearable device, and a computer-readable storage medium to improve the convenience of message processing operations of the smart wearable device.

[0004] In a first aspect, the present application provides a message processing method based on a smart wearable device, the method comprising:

[0005] Acquire the user's eye movement data and sight focus position, wherein the sight focus position is the position of the user's sight focus in the display interface of the smart wearable device;

[0006] Determining a positional relationship of the sight focus position relative to the extended screen based on the position of the extended screen in the display interface and the sight focus position, wherein the extended screen is used to display the notification message in the display interface;

[0007] The notification message is processed based on the eye movement data and / or the positional relationship.

[0008] Further, the processing of the notification message based on the position relationship includes:

[0009] When the sight focus position is located at the position of the extended screen, displaying the detailed content of the notification message on the extended screen;

[0010] When the visual focus position does not intersect with the position of the extended screen, the notification message is hidden or reduced in the extended screen.

[0011] Further, when the sight focus position is located at the position of the extended screen, displaying detailed content of the notification message on the extended screen includes:

[0012] When the sight focus position is located at the position of the extended screen, obtaining the residence time of the sight focus position in the extended screen;

[0013] When the stay time exceeds a preset time threshold, detailed content of the notification message is displayed in the extended screen.

[0014] Further, displaying detailed content of the notification message on the extended screen also includes:

[0015] Determine the priority of each message in the notification message based on the message source and / or the keyword, and determine the display order of each message in the notification message based on the priority of each message;

[0016] Based on the display order, each message in the notification message is displayed in the extended screen.

[0017] Further, the processing of the notification message based on the eye movement data includes:

[0018] Acquire the eye movement trajectory and eye movement sequence in the eye movement data;

[0019] Based on a preset mapping table, a target operation corresponding to the eye movement trajectory and the eye movement sequence is determined, and the target operation is performed on the notification message.

[0020] Furthermore, before determining the positional relationship of the sight focus position relative to the extended screen based on the position of the extended screen in the display interface and the sight focus position, the method further includes:

[0021] Based on the built-in camera of the smart wearable device, a terminal image of a target terminal is acquired, wherein the target terminal is a terminal that is in a detection area of ​​the smart wearable device and is in communication connection with the smart wearable device;

[0022] Calculating the position coordinates of the target terminal in the display interface based on the terminal image and the external parameters of the built-in camera relative to the world coordinate system;

[0023] Based on the location coordinates, the extended screen is rendered and generated in a preset area of ​​the target terminal.

[0024] Further, the calculating the position coordinates of the target terminal in the display interface based on the terminal image and the external parameters of the built-in camera relative to the world coordinate system includes:

[0025] Based on the terminal image, determining the terminal posture of the target terminal in the camera coordinate system;

[0026] Based on the external parameters of the built-in camera relative to the world coordinate system, convert the terminal posture into a rotation matrix and a translation vector in the world coordinate system;

[0027] Based on the rotation matrix, the translation vector and the center of the target terminal, the position coordinates of the target terminal in the display interface are calculated.

[0028] In a second aspect, the present application also provides a message processing device based on a smart wearable device, comprising:

[0029] An eye movement data acquisition module, used to acquire the user's eye movement data and sight focus position, wherein the sight focus position is the position of the user's sight focus in the display interface of the smart wearable device;

[0030] a position relationship acquisition module, used to determine the position relationship of the sight focus position relative to the extended screen based on the position of the extended screen in the display interface and the sight focus position, wherein the extended screen is used to display the notification message in the display interface;

[0031] A notification message processing module is used to process the notification message based on the eye movement data and / or the position relationship.

[0032] In a third aspect, the present application also provides a smart wearable device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the message processing method based on the smart wearable device as described above when executing the computer program.

[0033] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the message processing method based on the smart wearable device as described above.

[0034] The present application discloses a message processing method, device, smart wearable device and storage medium based on a smart wearable device, wherein the method obtains the user's eye movement data and the sight focus position, wherein the sight focus position is the position of the user's sight focus in the display interface of the smart wearable device; based on the position of the extended screen in the display interface and the sight focus position, the position relationship of the sight focus position relative to the extended screen is determined, wherein the extended screen is used to display a notification message in the display interface; based on the eye movement data and / or the position relationship, the notification message is processed. In the above manner, the present application implements the user's eye movement tracking based on the eye movement data and the sight focus position, and then processes the notification message based on the eye movement tracking situation, that is, based on the relative position relationship between the user's sight focus position and the extended screen used to display the notification message or the user's eye movement data. As a result, the present application implements notification message processing based on eye movement tracking, which not only simplifies the operation of the smart wearable device to process notification messages, improves the convenience of operation, but also is not easily disturbed by the external environment, improves the operation recognition rate, and further improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 is a schematic flow chart of a first embodiment of a message processing method based on a smart wearable device provided in an embodiment of the present application;

[0037] Figure 2 is a schematic flow chart of a second embodiment of a message processing method based on a smart wearable device provided in an embodiment of the present application;

[0038] Figure 3 is a rendering schematic diagram of a target terminal and an extended screen provided in an embodiment of the present application;

[0039] Figure 4 Schematic diagram of a message processing device based on a smart wearable device provided in an embodiment of the present application;

[0040] Figure 5 It is a schematic block diagram of the structure of a smart wearable device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0043] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0044] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0045] The embodiments of the present application provide a message processing method, apparatus, smart wearable device and storage medium based on a smart wearable device. The message processing method based on a smart wearable device can be applied to a server. The server can be an independent server or a server cluster.

[0046] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0047] See also Figure 1 , Figure 1 It is a schematic flow chart of a message processing method based on a smart wearable device provided in an embodiment of the present application.

[0048] like Figure 1 As shown, the message processing method based on the smart wearable device specifically includes steps S101 to S103.

[0049] S101, obtaining eye movement data and a sight focus position of a user, wherein the sight focus position is a position of the user's sight focus in a display interface of a smart wearable device;

[0050] The smart wearable device in this embodiment is a head-mounted smart wearable device, such as AR glasses, VR glasses and other head-mounted smart wearable devices; this embodiment is described by taking AR glasses as an example.

[0051] The current AR glasses still have many inconveniences in terms of message notifications. For example, users need to interact with AR glasses through gestures or voice in order to view and process notification messages. When interacting with gestures, users need to raise their hands to perform hand operations, which is not only cumbersome and easy to interfere with the user's current activities, but also easy to reduce the gesture recognition rate in low light conditions. When interacting with voice, the voice is not only easily affected by the surrounding noise, thereby reducing the voice recognition rate, but also inconvenient to operate in public places and other quiet environments.

[0052] In order to solve the above problems, this embodiment realizes the user's eye tracking based on eye movement data and the focus position of vision, and then realizes notification message processing based on the eye tracking situation. This not only simplifies the operation of processing notification messages of smart wearable devices and improves the convenience of operation, but also is not easily disturbed by the external environment, improves the operation recognition rate, and thus further improves the user experience.

[0053] Among them, eye movement data include eye movement trajectory, eye movement direction, eye movement sequence, pupil diameter, number and frequency of blinks, number and frequency of eye saccades, etc.; the line of sight focus position is the position of the user's line of sight focus in the display interface of the smart wearable device.

[0054] Specifically, the user's eye movement data is captured based on the perspective tracking sensor in the AR glasses, and the user's sight focus position is calculated based on the eye movement data and the pupil corneal reflection method (or other eye tracking algorithms). The sight focus position is the user's gaze point, which can be used to determine the position where the user's attention is distributed in the display interface.

[0055] In one embodiment, the view tracking sensor is a high-resolution camera that can quickly and accurately monitor parameters such as eye position and speed.

[0056] S102, determining a positional relationship of the sight focus position relative to the extended screen based on a position of the extended screen in the display interface and the sight focus position, wherein the extended screen is used to display a notification message in the display interface;

[0057] In this embodiment, the extended screen can be a functional component in the AR glasses, used to display notification messages of the user terminal associated with the AR glasses through the display interface of the AR glasses, so as to expand the notification messages of the user terminal to other areas in the user's field of view for display through the display interface of the AR glasses, avoiding the limitations of the small-screen display of the target terminal, so that the user can view the notification messages while maintaining awareness of the surrounding environment.

[0058] In one embodiment, the extended screen can also be used to display relevant functions of the smart wearable device so that the user can query and operate the relevant functions.

[0059] It can be understood that the extended screen is a high-performance cross-terminal development framework for smart large-screen devices, a fast application engine running on smart wearable devices, which is used to help developers quickly build application services for smart wearable devices and provide multi-screen interaction capabilities.

[0060] In this embodiment, the extended screen can be initially set at a preset position in the display interface, such as the upper left corner or the lower right corner; the extended screen can also be set in a preset area of ​​the detected user terminal; in a specific embodiment, the position of the extended screen in the display interface can be set according to the user setting operation.

[0061] Specifically, the sight focus position is compared with the position of the extended screen in the display interface to determine the relative position relationship between the sight focus position and the extended screen in the display interface.

[0062] The relative position relationship between the line of sight focus position and the extended screen in the display interface includes: the line of sight focus position coincides with the position of the extended screen, that is, the line of sight focus position is located within the position of the extended screen, and the line of sight focus position does not intersect with the position of the extended screen, which means that the user's line of sight focus position is in other areas of the display interface.

[0063] S103: Process the notification message based on the eye movement data and / or the position relationship.

[0064] In this embodiment, when it is analyzed based on eye movement data and / or the positional relationship of the line of sight focus relative to the extended screen that the user's attention has moved from other areas to the extended screen, it means that the user may have ended or paused other activities, and a notification message can be displayed on the extended screen so that the user can view or process the notification message.

[0065] Specifically, when the visual focus position coincides with the position of the extended screen, it means that the user's current attention is distributed on the extended screen, and the detailed content of the notification message can be expanded in the extended screen so that the user can view or process the notification message;

[0066] When the visual focus position does not intersect with the position of the extended screen, it means that the user's current attention is distributed in other areas outside the extended screen, and the extended screen can be folded or faded to reduce the visual interference of the extended screen and notification messages to the user.

[0067] Or when the eye movement trajectory in the eye movement data moves from other areas in the display interface to the extended screen, it is determined that the user's attention is about to be transferred to the location of the extended screen, and the detailed content of the notification message can be expanded in the extended screen;

[0068] When the eye movement trajectory moves from the extended screen to other areas in the display interface, it is determined that the user's attention is about to be transferred to other areas, and the extended screen can be folded or faded.

[0069] Or when the eye movement track in the eye movement data moves from other areas in the display interface to the extended screen and the sight focus position coincides with the position of the extended screen, the detailed content of the notification message can be expanded in the extended screen;

[0070] When the eye movement trajectory moves from the extended screen to other areas in the display interface and the sight focus position does not intersect with the position of the extended screen, the extended screen can be retracted or faded.

[0071] This embodiment provides a message processing method based on a smart wearable device, the method obtains the user's eye movement data and the sight focus position, wherein the sight focus position is the position of the user's sight focus in the display interface of the smart wearable device; based on the position of the extended screen in the display interface and the sight focus position, determine the positional relationship of the sight focus position relative to the extended screen, wherein the extended screen is used to display the notification message in the display interface; based on the eye movement data and / or the positional relationship, process the notification message. In the above manner, the present application implements the user's eye tracking based on the eye movement data and the sight focus position, and then processes the notification message based on the eye tracking situation, that is, based on the relative positional relationship between the user's sight focus position and the extended screen used to display the notification message or the user's eye movement data. Therefore, the present application implements notification message processing based on eye tracking, which not only simplifies the operation of the smart wearable device to process notification messages and improves the convenience of operation, but also is not easily disturbed by the external environment, improves the operation recognition rate, and further improves the user experience.

[0072] See also Figure 2 , Figure 2 It is a schematic flow chart of a message processing method based on a smart wearable device provided in an embodiment of the present application.

[0073] like Figure 2As shown, the message processing method based on the smart wearable device further includes, before step S102:

[0074] Step S104, acquiring a terminal image of a target terminal based on a built-in camera of the smart wearable device, wherein the target terminal is a terminal that is in a detection area of ​​the smart wearable device and is in communication connection with the smart wearable device;

[0075] Step S105, calculating the position coordinates of the target terminal in the display interface based on the terminal image and the external parameters of the built-in camera relative to the world coordinate system;

[0076] Step S106: Based on the position coordinates, the extended screen is rendered and generated in a preset area of ​​the target terminal.

[0077] The step of calculating the position coordinates of the target terminal in the display interface based on the terminal image and the external parameters of the built-in camera relative to the world coordinate system includes:

[0078] Based on the terminal image, determining the terminal posture of the target terminal in the camera coordinate system;

[0079] Based on the external parameters of the built-in camera relative to the world coordinate system, convert the terminal posture into a rotation matrix and a translation vector in the world coordinate system;

[0080] Based on the rotation matrix, the translation vector and the center of the target terminal, the position coordinates of the target terminal in the display interface are calculated.

[0081] In this embodiment, the notification message can be a notification message in the AR glasses, or a notification message in a terminal (such as a mobile phone, an i pad, a smart bracelet) that is connected to the AR glasses for communication. The terminal that receives the notification message is used as the target terminal; or based on the built-in camera in the AR glasses, the terminal that the user is viewing or using is detected as the target terminal. Based on the built-in camera, the terminal image of the target terminal is captured, and based on the terminal image, the rectangular frame coordinates (x1, y1, x2, y2) of the target terminal on the image plane are determined. Based on the rectangular frame coordinates (x1, y1, x2, y2) and the camera internal parameters, the size (width w_img and height h_img) of the target terminal in the image coordinate system is calculated.

[0082] Among them, the camera intrinsic parameters include the focal length (f_x, f_y), principal point coordinates (c_x, c_y) and distortion parameters used to describe the camera's optical characteristics and imaging model.

[0083] It can be understood that the size of the target terminal is calculated based on perspective transformation or similar triangle principle, and corresponding correction operations may be performed to prevent lens distortion.

[0084] Based on PnP (Perspective-n-Point) and combined with the four vertices corresponding to the rectangular box of the target terminal (or the feature points corresponding to the target terminal) and the depth information corresponding to the target terminal, the rotation matrix R_phone and the translation vector t_phone of the target terminal in the camera coordinate system are calculated.

[0085] In one embodiment, the depth information of the target terminal can be acquired in advance through a depth camera or structured light; if it cannot be acquired, the depth information of the target terminal can be estimated based on multi-view geometry, SLAM (simultaneous localization and mapping) or additional hypothesis method.

[0086] Based on the rotation matrix R_phone and translation vector t_phone of the target terminal in the camera coordinate system, the position and posture (R_phone, t_phone) of the target terminal in the camera coordinate system can be determined, and then based on the known external parameters (R_cam, T_cam) of the built-in camera relative to the world coordinate system, the position and posture (R_phone, t_phone) of the target terminal in the camera coordinate system is converted to the world coordinate system, wherein the rotation matrix and translation vector of the target terminal in the world coordinate system are respectively:

[0087] R_work=R_cam*R_phone;

[0088] T_wor ld=R_cam*t_phone+T_cam.

[0089] Calculate the position of the center of the target terminal (such as the average value of the four corner points of the target terminal in the world coordinate system), and determine the directions of the three axes of length, width and height of the target terminal in the world coordinate system through the rotation matrix R_phone of the target terminal in the camera coordinate system and the known physical dimensions of the target terminal (the length, width and height of the target terminal).

[0090] It is understandable that the target terminal edge line or other landmarks may be used to assist in determining the above directions.

[0091] A unit coordinate system (xphone, yphone, zphone) is constructed with the center of the target terminal as the origin and aligned along the length, width and height of the target terminal, so as to map the detected target terminal to a real three-dimensional space (i.e., the position coordinates in the display interface).

[0092] like Figure 3 As shown, the origin position of the unit coordinate system is used as the position coordinate of the target terminal in the display interface. Based on the position coordinate, the position of the target terminal in the display interface is determined ((100.10), (150,10), (100,110), (150,110)), and at the position coordinate, an extended screen is rendered (such as Figure 3 The first extended screen component shown in FIG. 1 may be located in a preset area (such as a first extended screen component) of the detected target terminal. Figure 3 The extended screen is shown on the left side of the screen.

[0093] It can be understood that the extended screen may include one or more areas. As shown in the figure, a first extended screen component corresponding to one of the extended screen areas is rendered on the left side of the mobile terminal, and a second extended screen component can be rendered on the right side, etc.

[0094] The present embodiment provides a message processing method based on a smart wearable device, which uses the display interface of the AR glasses to expand the notification messages received by the terminal that is in communication with the AR glasses and is in the detection area of ​​the AR glasses to other areas in the user's field of view, thereby avoiding the limitation of the small screen display of the target terminal, so that the user can view the notification messages while maintaining awareness of the surrounding environment. In addition, current AR glasses often use fixed positions and formats when displaying message notifications, lack flexibility and personalized settings, and cannot meet the needs of users in different scenarios. In the present embodiment, the position of the target terminal is determined on the display interface of the AR glasses, and an extended screen is rendered around the target terminal. The notification message can move with the movement of the target terminal, which improves the intuitiveness and naturalness of the notification message display effect, thereby further improving the user experience.

[0095] Furthermore, the message processing method based on the smart wearable device includes in step S103:

[0096] When the sight focus position is located at the position of the extended screen, displaying the detailed content of the notification message on the extended screen;

[0097] When the visual focus position does not intersect with the position of the extended screen, the notification message is hidden or reduced in the extended screen.

[0098] In this embodiment, based on the line of sight focus position, the notification message is displayed only when it is determined that the user's attention is on the extended screen, thereby avoiding the notification message interfering with the user's line of sight when the user is performing other activities. The line of sight focus position will change with the change of the user's attention. Therefore, the area where the user's attention is located can be determined by the line of sight focus position. When the line of sight focus position is located at the position of the extended screen, it means that the current user's attention is distributed at the position of the extended screen displaying the notification message. At this time, the detailed content of the notification message can be displayed in the extended screen so that the user can view or process the notification message. When the line of sight focus position does not intersect with the position of the extended screen, it means that the user's line of sight focus position is in other areas of the display interface, and the notification message is hidden or reduced in the extended screen.

[0099] In one embodiment, the movement trajectory of the line of sight focus position can be rendered in the display interface of the AR glasses, such as rendering the line of sight focus position as a mouse arrow, a red dot or other mark in the display interface, so that the user can understand the movement of the line of sight focus position in the display interface.

[0100] Further, when the sight focus position is located at the position of the extended screen, displaying detailed content of the notification message on the extended screen includes:

[0101] When the sight focus position is located at the position of the extended screen, obtaining the residence time of the sight focus position in the extended screen;

[0102] When the stay time exceeds a preset time threshold, detailed content of the notification message is displayed in the extended screen.

[0103] In this embodiment, the residence time of the focus position on the extended screen is further combined to avoid misjudgment (such as when the user's line of sight only passes through the extended screen area). When the line of sight focus position is located at the position of the extended screen and the residence time exceeds the preset time threshold, the detailed content of the notification message is displayed in the extended screen. When the residence time does not exceed the preset time threshold, a prompt message of whether to expand the extended screen can be generated, so that when a confirmation message from the user is received, the detailed content of the notification message is displayed in the extended screen.

[0104] Further, displaying detailed content of the notification message on the extended screen also includes:

[0105] Determine the priority of each message in the notification message based on the message source and / or the keyword, and determine the display order of each message in the notification message based on the priority of each message;

[0106] Based on the display order, each message in the notification message is displayed in the extended screen.

[0107] In this embodiment, when the number of displayed messages or words of the notification message is large, the priority of each message in the notification message can be further determined based on the message source (such as social media, email, instant messaging tool, etc.) or keywords (such as sender, subject, etc.) of the notification message. For example, the notification message corresponding to the instant messaging tool or the notification message from the sender who is focused in advance is set to a high priority, and the notification message from the group chat or the notification message of the unfocused contact is set to a low priority.

[0108] Based on the priority of each message, the display order of each message is determined, and then based on the display order, each message is displayed from the earliest to the latest.

[0109] Through the above method, users can quickly view urgent or important notification messages, thereby improving user experience.

[0110] Further, the processing of the notification message based on the position relationship includes:

[0111] Acquire the eye movement trajectory and eye movement sequence in the eye movement data;

[0112] Based on a preset mapping table, a target operation corresponding to the eye movement trajectory and the eye movement sequence is determined, and the target operation is performed on the notification message.

[0113] In this embodiment, the eye movement data includes eye movement trajectory, eye movement direction, eye movement sequence, pupil diameter, blinking number and frequency, eye saccade number and frequency. The user can set different eye movement trajectories and eye movement sequences in the mapping table in advance, corresponding to different operations, so that the user can mark the message as read or quickly preview the message list through simple eye movements, such as gaze retention, staring or scanning. The notification message is processed; or the user's eye movement trajectory and eye movement sequence detected are compared with the mapping table. And the target operation corresponding to the eye movement trajectory and eye movement sequence is determined, and the target operation is performed on the notification message.

[0114] In one embodiment, the user can execute notification message processing operations such as replying, forwarding, and deleting messages by setting eye movement instructions such as blinking frequency or specific eye movement sequences, thereby improving the user's efficiency in processing notification information while meeting the user's needs to reduce manual operations.

[0115] Furthermore, the ambient light sensor and head posture sensor built into the AR glasses can be used to detect the lighting conditions around the terminal and the user's head posture. Based on the pre-set processing operations, the detected ambient light conditions and the user's head posture, the brightness, contrast and viewing angle of the notification message can be dynamically adjusted to improve the clarity of the notification message in various environments and the comfort of the user's viewing.

[0116] See also Figure 4 , Figure 4 The embodiment of the present application provides a schematic block diagram of a message processing device based on a smart wearable device, wherein the message processing device based on a smart wearable device is used to execute the aforementioned message processing method based on a smart wearable device. The message processing device based on a smart wearable device can be configured on a server.

[0117] like Figure 4 As shown, the message processing device 400 based on the smart wearable device includes:

[0118] The eye movement data acquisition module 401 is used to acquire the user's eye movement data and the sight focus position, wherein the sight focus position is the position of the user's sight focus in the display interface of the smart wearable device;

[0119] A position relationship acquisition module 402, configured to determine a position relationship of the sight focus position relative to the extended screen based on the position of the extended screen in the display interface and the sight focus position, wherein the extended screen is used to display the notification message in the display interface;

[0120] The notification message processing module 403 is used to process the notification message based on the eye movement data and / or the position relationship.

[0121] Furthermore, the notification message processing module 403 includes:

[0122] A message display unit, configured to display detailed content of the notification message on the extended screen when the sight focus position is located at the position of the extended screen;

[0123] A message hiding unit is used to hide or reduce the notification message in the extended screen when the visual focus position does not intersect with the position of the extended screen.

[0124] Furthermore, the message display unit is also used for:

[0125] When the sight focus position is located at the position of the extended screen, obtaining the residence time of the sight focus position in the extended screen;

[0126] When the stay time exceeds a preset time threshold, detailed content of the notification message is displayed in the extended screen.

[0127] Furthermore, the message display unit is also used for:

[0128] Determine the priority of each message in the notification message based on the message source and / or the keyword, and determine the display order of each message in the notification message based on the priority of each message;

[0129] Based on the display order, each message in the notification message is displayed in the extended screen.

[0130] Furthermore, the notification message processing module 403 includes:

[0131] A message operation unit, used for obtaining the eye movement trajectory and eye movement sequence in the eye movement data;

[0132] Based on a preset mapping table, a target operation corresponding to the eye movement trajectory and the eye movement sequence is determined, and the target operation is performed on the notification message.

[0133] Furthermore, the message processing device 400 based on the smart wearable device also includes an extended screen rendering module, which is used to:

[0134] Based on the built-in camera of the smart wearable device, a terminal image of a target terminal is acquired, wherein the target terminal is a terminal that is in a detection area of ​​the smart wearable device and is in communication connection with the smart wearable device;

[0135] Calculating the position coordinates of the target terminal in the display interface based on the terminal image and the external parameters of the built-in camera relative to the world coordinate system;

[0136] Based on the location coordinates, the extended screen is rendered and generated in a preset area of ​​the target terminal.

[0137] Furthermore, the extended screen rendering module is also used for:

[0138] Based on the terminal image, determining the terminal posture of the target terminal in the camera coordinate system;

[0139] Based on the external parameters of the built-in camera relative to the world coordinate system, convert the terminal posture into a rotation matrix and a translation vector in the world coordinate system;

[0140] Based on the rotation matrix, the translation vector and the center of the target terminal, the position coordinates of the target terminal in the display interface are calculated.

[0141] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0142] The above-mentioned device can be implemented in the form of a computer program. Figure 5 The system runs on the smart wearable device shown.

[0143] See also Figure 5 , Figure 5 1 is a schematic block diagram of the structure of a smart wearable device provided in an embodiment of the present application. The smart wearable device may be a server.

[0144] See also Figure 5 The smart wearable device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0145] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any message processing method based on the smart wearable device.

[0146] The processor is used to provide computing and control capabilities to support the operation of the entire smart wearable device.

[0147] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any message processing method based on the smart wearable device.

[0148] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the smart wearable device to which the scheme of the present application is applied. The specific smart wearable device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0149] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0150] The processor is used to run a computer program stored in the memory to implement any step in the above-mentioned message processing method based on a smart wearable device.

[0151] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any message processing method based on a smart wearable device provided in an embodiment of the present application.

[0152] The computer-readable storage medium may be an internal storage unit of the smart wearable device described in the foregoing embodiment, such as a hard disk or memory of the smart wearable device. The computer-readable storage medium may also be an external storage device of the smart wearable device, such as a plug-in hard disk, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash memory card (Flash Card), etc., equipped on the smart wearable device.

[0153] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A message processing method based on a smart wearable device, characterized in that: The method comprises: Acquire the user's eye movement data and sight focus position, wherein the sight focus position is the position of the user's sight focus in the display interface of the smart wearable device; Determining a positional relationship of the sight focus position relative to the extended screen based on the position of the extended screen in the display interface and the sight focus position, wherein the extended screen is used to display the notification message in the display interface; The notification message is processed based on the eye movement data and / or the positional relationship.

2. The message processing method based on the smart wearable device according to claim 1, characterized in that: The processing of the notification message based on the position relationship includes: When the sight focus position is located at the position of the extended screen, displaying the detailed content of the notification message on the extended screen; When the visual focus position does not intersect with the position of the extended screen, the notification message is hidden or reduced in the extended screen.

3. The message processing method based on the smart wearable device according to claim 2 is characterized in that: When the sight focus position is located at the position of the extended screen, displaying detailed content of the notification message on the extended screen includes: When the sight focus position is located at the position of the extended screen, obtaining the residence time of the sight focus position in the extended screen; When the stay time exceeds a preset time threshold, detailed content of the notification message is displayed in the extended screen.

4. The message processing method based on the smart wearable device according to claim 2, characterized in that: The displaying detailed content of the notification message on the extended screen further includes: Determine the priority of each message in the notification message based on the message source and / or the keyword, and determine the display order of each message in the notification message based on the priority of each message; Based on the display order, each message in the notification message is displayed in the extended screen.

5. The message processing method based on the smart wearable device according to claim 1, characterized in that: The processing of the notification message based on the eye movement data includes: Acquire the eye movement trajectory and eye movement sequence in the eye movement data; Based on a preset mapping table, a target operation corresponding to the eye movement trajectory and the eye movement sequence is determined, and the target operation is performed on the notification message.

6. The message processing method based on a smart wearable device according to any one of claims 1 to 5, characterized in that: Before determining the positional relationship of the sight focus position relative to the extended screen based on the position of the extended screen in the display interface and the sight focus position, the method further includes: Based on the built-in camera of the smart wearable device, a terminal image of a target terminal is acquired, wherein the target terminal is a terminal that is in a detection area of ​​the smart wearable device and is in communication connection with the smart wearable device; Calculating the position coordinates of the target terminal in the display interface based on the terminal image and the external parameters of the built-in camera relative to the world coordinate system; Based on the location coordinates, the extended screen is rendered and generated in a preset area of ​​the target terminal.

7. The message processing method based on the smart wearable device according to claim 6, characterized in that: The calculating the position coordinates of the target terminal in the display interface based on the terminal image and the external parameters of the built-in camera relative to the world coordinate system includes: Based on the terminal image, determining the terminal posture of the target terminal in the camera coordinate system; Based on the external parameters of the built-in camera relative to the world coordinate system, convert the terminal posture into a rotation matrix and a translation vector in the world coordinate system; Based on the rotation matrix, the translation vector and the center of the target terminal, the position coordinates of the target terminal in the display interface are calculated.

8. A message processing device based on a smart wearable device, characterized in that: include: An eye movement data acquisition module, used to acquire the user's eye movement data and sight focus position, wherein the sight focus position is the position of the user's sight focus in the display interface of the smart wearable device; a position relationship acquisition module, used to determine the position relationship of the sight focus position relative to the extended screen based on the position of the extended screen in the display interface and the sight focus position, wherein the extended screen is used to display the notification message in the display interface; A notification message processing module is used to process the notification message based on the eye movement data and / or the position relationship.

9. A smart wearable device, characterized in that: The smart wearable device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the message processing method based on the smart wearable device as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the message processing method based on a smart wearable device according to any one of claims 1 to 7.