Information processing method and device, intelligent glasses, computer equipment and medium

Abstract

The invention relates to the field of data processing, in particular to an information processing method and device, intelligent glasses, computer equipment and a medium, a first user wears the intelligent glasses and an intelligent watch, and first voice information sent by the first user is received; second voice information sent by the smart watch is received, and the voice information is input to the smart watch by a second user; acquiring action information of the second user through an image sensor; and performing identification processing on the first voice information, the second voice information and the action information, and forming a communication dialog box on a display interface of the intelligent glasses. By adopting the method, the dialogue communication content of the user is accurately and effectively identified, so that the user can accurately know the dialogue communication content.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to an information processing method, device, smart glasses, computer equipment and medium. Background Art

[0002] Smart glasses, as a wearable device that combines the functions of traditional glasses with modern smart devices, have made significant technological progress in recent years. Equipped with advanced technologies such as micro displays, cameras, sensors, etc., they can provide users with a variety of functions such as information display, augmented reality (AR), photo taking, video recording, and navigation. Smart watches, with their low power consumption and easy portability, provide accurate data and guidance in sports and health monitoring. With the rapid development of the Internet of Things and wireless communication technologies, data interaction and collaborative work between smart glasses and smart watches have become possible. This collaborative work can not only enhance the user's interactive experience, but also play an important role in multi-user communication scenarios.

[0003] Normally, when a user wears smart glasses and a smart watch, the smart watch usually collects the voice information of the user or the other party through its built-in microphone or other voice collection devices, and then recognizes the voice information and sends the recognized text content to the smart glasses for display to the user to assist the user in conversation.

[0004] However, the study found that since the transmission and collection of voice information will be affected by environmental factors, if the user is in a noisy environment and is communicating with the other party, the voice information during the conversation cannot be accurately collected by the smart watch, which will reduce the accuracy of the recognized text content, and further reduce the effectiveness and accuracy of the conversation content received by the user through the smart glasses. In addition, in the process of people's conversations, they often do not rely entirely on language to communicate, but may also express themselves through other non-verbal forms of expression such as body movements and facial expressions. If only the voice information of the user or the other party is collected by the voice collection device for information display, the opinions expressed by the other party through body movements may be omitted, which will lead to omissions or misunderstandings of the conversation content, thereby reducing the effectiveness and accuracy of the recognition of the conversation content. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide an information processing method, apparatus, smart glasses, computer equipment and medium to accurately and effectively identify the content of user conversations and exchanges, so that users can accurately know the content of conversations and exchanges.

[0006] In a first aspect, an embodiment of the present application provides an information processing method, which is applied to smart glasses, and a first user wears the smart glasses and a smart watch, and the method includes: Receiving a first voice message sent by a first user; receiving a second voice message sent by the smart watch, wherein the voice message is input to the smart watch by a second user; acquiring the action information of the second user through an image sensor; The first voice information, the second voice information and the action information are recognized and processed, and a communication dialog box is formed on the display interface of the smart glasses.

[0007] Optionally, the first voice information, the second voice information and the action information are recognized and processed to form a communication dialog box on the display interface of the smart glasses, including: The semantics corresponding to the first voice information, the second voice information and the action information are arranged in chronological order and semantic order to form the communication dialog box.

[0008] Optionally, the first voice information, the second voice information and the action information are recognized and processed to form a communication dialog box on the display interface of the smart glasses, including: Determining whether a time interval of the second voice information coincides with a time interval of the action information; In response to the time intervals of the second voice information and the action information not overlapping, the semantic information corresponding to the action information is determined as the content in the communication dialog box.

[0009] Optionally, the first voice information, the second voice information and the action information are recognized and processed to form a communication dialog box on the display interface of the smart glasses, including: In response to the time interval of the second voice information and the action information being coincident, determining an approximate value of the second voice information and the action information; In response to the approximation value being greater than a preset value, the semantic content corresponding to the second voice information is determined as the content in the communication dialog box.

[0010] Optionally, the first voice information, the second voice information and the action information are recognized and processed to form a communication dialog box on the display interface of the smart glasses, including: In response to the time interval of the second voice information and the action information being coincident, determining an approximate value of the second voice information and the action information; In response to the approximation value being less than a preset value, determining a first semantic content corresponding to the second voice information, and determining a second semantic content corresponding to the action information; The first semantic content and the second semantic content are fused, and the fused semantic information is determined as the content in the communication dialog box.

[0011] Optionally, after receiving the first voice message sent by the first user, the method further includes: The first voice information is translated and sent to the smart watch, so that the smart watch displays or plays the translated first voice information.

[0012] Optionally, the method further includes: determining the intention of the first user in the communication dialog box, and determining a strategy for prompting the first user according to the intention of the first user.

[0013] Optionally, determining a strategy for prompting the first user according to the intention of the first user includes: determining whether the policy is associated with the smart watch; If the policy is related to a smart watch, the policy is sent to the smart watch.

[0014] In a second aspect, an embodiment of the present application provides an information processing device, which is applied to smart glasses. A first user wears the smart glasses and a smart watch. The device includes: A first voice information receiving module, used to receive a first voice information sent by a first user; A second voice information receiving module, configured to receive a second voice information sent by the smart watch, wherein the voice information is input by a second user to the smart watch; an action information acquisition module, configured to acquire the action information of the second user through an image sensor; The communication dialog box display module is used to identify and process the first voice information, the second voice information and the action information, and form a communication dialog box on the display interface of the smart glasses.

[0015] In a third aspect, a pair of smart glasses are provided, wherein when the smart glasses are in operation, the steps of the information processing method described in any optional implementation manner in the first aspect are executed.

[0016] In a fourth aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the information processing method described in any one of the optional implementations of the first aspect are performed.

[0017] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the information processing method described in any optional implementation manner in the above-mentioned first aspect are executed.

[0018] The technical solution provided by this application includes but is not limited to the following beneficial effects: When the first user wears smart glasses and a smart watch, the present application receives the first voice information sent by the first user through the smart glasses, and receives the second voice information sent by the smart watch, wherein the voice information is input by the second user to the smart watch, and the voice information of both parties in the dialogue can be effectively collected in real time. Then, the motion information of the second user is obtained through the image sensor, and the body movements and other information of the dialogue party can be collected in real time, providing multi-dimensional and multi-format data support for the subsequent recognition of the content of the dialogue. Finally, the first voice information, the second voice information and the motion information are recognized and processed, and a communication dialog box is formed on the display interface of the smart glasses. By comprehensively considering the two types of multi-dimensional and multi-format information of voice information and motion information for recognition to form a communication dialog box, the comprehensiveness of information recognition can be improved, and more accurate and effective recognition results can be obtained, thereby realizing accurate and effective recognition of the content of user dialogue and communication, so that the user can accurately know the content of the dialogue and communication.

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flow chart of an information processing method provided by an embodiment of the present invention is shown; Figure 2A flow chart of a method for forming a communication dialog box provided by an embodiment of the present invention is shown; Figure 3 A flowchart showing a specific communication dialog method provided by an embodiment of the present invention is shown; Figure 4 A flowchart showing a second specific method for forming a communication dialog box provided by an embodiment of the present invention; Figure 5 A flow chart of a method for determining a user prompt strategy provided by an embodiment of the present invention is shown; Figure 6 A flowchart of a method for translating and responding in real time based on smart glasses and watches provided by an embodiment of the present invention is shown; Figure 7 A schematic diagram showing the structure of an information processing device provided by an embodiment of the present invention is shown; Figure 8 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0023] To facilitate understanding of this application, Figure 1 The flowchart of an information processing method provided by an embodiment of the present invention is shown to describe the contents of the embodiment of the present application in detail.

[0024] See also Figure 1 As shown, Figure 1 A flowchart of an information processing method provided by an embodiment of the present invention is shown, wherein the method is applied to smart glasses, and a first user wears the smart glasses and a smart watch, wherein the method comprises steps S101 to S104: S101: receiving a first voice message sent by a first user.

[0025] Specifically, the first user sends voice information through the built-in microphone of the smart glasses or other voice input devices.

[0026] S102: Receive a second voice message sent by the smart watch, wherein the voice message is input to the smart watch by a second user.

[0027] Specifically, the second user inputs voice information through the microphone of the smart watch, and the information is then sent to the smart glasses.

[0028] S103: Acquire the action information of the second user through an image sensor.

[0029] Specifically, an image sensor (such as a camera) of the smart glasses captures the actions of the second user, such as gestures, expressions, etc.

[0030] S104: Recognize and process the first voice information, the second voice information, and the action information, and form a communication dialog box on the display interface of the smart glasses.

[0031] Specifically, the processor of the smart glasses performs voice recognition on the received first voice information and second voice information. At the same time, it performs motion recognition on the motion information acquired by the image sensor. The recognition results are integrated to form a communication dialog box containing text, voice and motion information, which is displayed on the display interface of the smart glasses.

[0032] This application, through steps S101 to S104, when the first user wears smart glasses and a smart watch, receives the first voice information sent by the first user through the smart glasses, and receives the second voice information sent by the smart watch, wherein the voice information is input by the second user to the smart watch, and the voice information of both parties in the dialogue can be effectively collected in real time. Then, the action information of the second user is obtained through the image sensor, and the body movements and other information of the dialogue party can be collected in real time, providing multi-dimensional and multi-format data support for the subsequent dialogue content recognition. Finally, the first voice information, the second voice information and the action information are recognized and processed, and a communication dialog box is formed on the display interface of the smart glasses. By comprehensively considering the two types of multi-dimensional and multi-format information of voice information and action information for recognition to form a communication dialog box, the comprehensiveness of information recognition can be improved, and more accurate and effective recognition results can be obtained, thereby realizing accurate and effective recognition of the content of user dialogue and communication, so that the user can accurately know the content of the dialogue and communication.

[0033] In an optional implementation, the first voice information, the second voice information and the action information are recognized and processed, and a communication dialog box is formed on the display interface of the smart glasses, including: The semantics corresponding to the first voice information, the second voice information and the action information are arranged in chronological order and semantic order to form the communication dialog box.

[0034] Specifically, when processing the first voice information, the second voice information and the action information to form a communication dialog box on the smart glasses, it is indeed necessary to arrange the information in a time and semantic order. For voice information, a voice recognition algorithm is used to convert the voice into voice information. For action information, an action recognition algorithm is used to convert the image data corresponding to the action information into an understandable action description (such as "nod", "wave", etc.). According to the order of the semantics corresponding to the voice information and the semantics corresponding to the action description, the voice information and action information are arranged from first to last to form a communication dialog box.

[0035] In some preferred embodiments, the display of action descriptions in the communication dialog box can be expressions or emoticons, and the present application does not impose any particular restrictions on this. When the voice information and action information are arranged from first to last to form a communication dialog box according to the order of the semantics corresponding to the voice information and the semantics corresponding to the action description, first, their basic order is determined according to the timestamp of each message, and a preliminary timeline is provided for subsequent adjustment of the semantic order. The identified voice text and action description are semantically analyzed to identify key elements in the conversation, such as questions, answers, instructions, and responses. According to the results of the semantic analysis, the preliminary timeline is adjusted to ensure the fluency and logic of the conversation, so that the information is arranged in the order of natural conversation.

[0036] Arrange the recognized voice text and action icons in the dialog box according to the natural flow of the conversation based on the order of time and semantics. When new information is added, the dialog box should automatically scroll or adjust the layout to accommodate the information.

[0037] In the above steps, the semantics corresponding to the first voice information, the second voice information and the action information are arranged in chronological order and semantic order to form a communication dialog box, which helps to enhance the coherence of the dialogue, improve the user experience and promote effective communication.

[0038] In an alternative embodiment, see Figure 2 As shown, Figure 2 A flow chart of a method for forming a communication dialog box provided by an embodiment of the present invention is shown, wherein the first voice information, the second voice information and the action information are recognized and processed, and a communication dialog box is formed on the display interface of the smart glasses, including steps S201-S202: S201: Determine whether the time interval of the second voice information coincides with the time interval of the action information.

[0039] Specifically, during a conversation, the parties may use both body movements and language, or may only use body movements or head movements and remain silent. In order to conduct targeted analysis of the above different scenarios, when processing the first voice information, the second voice information and the action information to form a communication dialog box on the smart glasses, it is indeed necessary to judge the time relationship between these information and determine whether they should be used as part of the conversation content, so as to adapt different information display strategies for different scenarios.

[0040] For each piece of information (including the first voice information, the second voice information and the action information), obtain its start time and end time, and determine its time interval based on its start time and end time. Determine whether the time interval of the second voice information overlaps with the time interval of the action information to determine whether the two time intervals have an intersection.

[0041] S202: In response to the second voice information and the action information not having a time interval overlap, determining the semantic information corresponding to the action information as the content in the communication dialog box.

[0042] Specifically, if the time intervals of the second voice information and the action information do not overlap, that is, they do not occur at the same time, then the semantic information corresponding to the action information shall prevail and be determined separately as the content in the communication dialog box.

[0043] In some specific embodiments of the present application, based on computer vision technology, for example, through 3D convolutional neural networks (3D CNNs) or a skeleton key point recognition algorithm (ST-GCN), etc., the action corresponding to the action information is identified, and then the semantic information corresponding to the action information is determined through the action.

[0044] The present application captures and processes voice information and motion information in real time through steps S201~S202, and adjusts the content displayed on the display interface of the smart glasses based on the overlap of the time intervals of the voice information and the motion information, which can improve the efficiency and accuracy of communication and enhance the user experience.

[0045] In an alternative embodiment, see Figure 3 As shown, Figure 3 A flowchart of a specific communication dialog method provided by an embodiment of the present invention is shown, wherein the first voice information, the second voice information and the action information are recognized and processed, and a communication dialog is formed on the display interface of the smart glasses, including steps S301-S302: S301: In response to the time interval of the second voice information and the action information being coincident, determine an approximate value of the second voice information and the action information.

[0046] Specifically, when it is detected that the time intervals of the second voice information and the action information overlap, that is, the user is performing actions while speaking, the semantic information corresponding to the action information is identified based on computer vision technology, and the semantic information corresponding to the second voice information is determined by a speech recognition algorithm. Then, the approximate value of the semantic information corresponding to the second voice information and the action information is calculated based on a pre-configured approximation algorithm, and the approximate value can reflect the correlation between the two.

[0047] In some specific implementations, a computational method based on semantic understanding is used to perform semantic recognition. Specifically, the semantic approximation may be calculated based on a semantic dictionary or a computational method based on a semantic expression.

[0048] S302: In response to the approximation value being greater than a preset value, the semantic content corresponding to the second voice information is determined as the content in the communication dialog box.

[0049] Specifically, if the approximate value is greater than a preset value, it indicates that there is a strong correlation between the second voice information and the action information. In this case, the semantic content corresponding to the second voice information is determined as the content in the communication dialog box.

[0050] This application simultaneously considers the overlap of the time intervals of the second voice information and the action information through steps S301~S302, and calculates the approximate value between them to determine the content in the communication dialog box, which can improve the accuracy of communication understanding, enhance the user experience, and help promote barrier-free communication.

[0051] In an alternative embodiment, see Figure 4 As shown, Figure 4 A flowchart of a second specific method for forming a communication dialog box provided by an embodiment of the present invention is shown, wherein the first voice information, the second voice information and the action information are recognized and processed, and a communication dialog box is formed on the display interface of the smart glasses, including steps S401 to S403: S401: In response to the time interval of the second voice information and the action information being coincident, determining approximate values ​​of the second voice information and the action information.

[0052] Specifically, when it is detected that the time intervals of the second voice information and the action information overlap, that is, the user is performing actions while speaking, the semantic information corresponding to the action information is identified based on computer vision technology, and the semantic information corresponding to the second voice information is determined by a speech recognition algorithm. Then, the approximate value of the semantic information corresponding to the second voice information and the action information is calculated based on a pre-configured approximation algorithm, and the approximate value can reflect the correlation between the two.

[0053] In some specific implementations, a computational method based on semantic understanding is used to perform semantic recognition. Specifically, the semantic approximation may be calculated based on a semantic dictionary or a computational method based on a semantic expression.

[0054] S402: In response to the approximation value being less than a preset value, determining a first semantic content corresponding to the second voice information, and determining a second semantic content corresponding to the action information.

[0055] Specifically, if the approximation value is less than the preset value, it indicates that the correlation or synchronization between the second voice information and the action information is not strong, or the information they convey is independent. In this case, the first semantic content corresponding to the second voice information and the second semantic content corresponding to the action information are determined respectively.

[0056] S403: Fusing the first semantic content and the second semantic content, and determining the fused semantic information as the content in the communication dialog box.

[0057] Specifically, the first semantic content and the second semantic content are fused to combine the voice information and the action information at the semantic level. Combining the two information sources of voice and action makes the fused semantic information more accurately reflect the user's communication intention and can more comprehensively express the user's intention or communication content.

[0058] The first semantic content and the second semantic content are integrated, including: extracting key features from the first semantic content and the second semantic content respectively, the key features including vocabulary, phrases, grammatical structures, action types, and action parameters. The extracted features are integrated using a feature fusion algorithm (such as weighted average, maximum pooling, attention mechanism, etc.) to obtain integrated semantic information.

[0059] The present application simultaneously considers the time interval overlap and approximate value of the second voice information and the action information through steps S401 to S403, and determines and integrates the semantic content accordingly, which can more accurately understand the user's communication intention and improve the performance of the smart glasses system in communication understanding, user experience, and diversity of communication methods.

[0060] In an optional implementation, after receiving the first voice message sent by the first user, the method further includes: The first voice information is translated and sent to the smart watch, so that the smart watch displays or plays the translated first voice information.

[0061] Specifically, the recognized text is translated from the original language into a target language that the smartwatch user may understand. Machine translation technology, such as a neural machine translation model based on deep learning, can be used for translation. After obtaining the translated result, the translated text message is sent to the smartwatch. After the smartwatch receives the translated message, it can choose to display the text message on the screen or play the translated voice message through TTS text-to-speech technology according to the settings or user preferences.

[0062] Through the above steps, the first voice message is translated and sent to the smart watch, and the translated message is displayed or played on the smart watch, which can provide users with instant translation and communication capabilities and help users achieve cross-language communication; in addition, as a personal device, the smart watch can instantly receive and display translated information, providing users with a convenient way to obtain information. Users do not need to frequently check their phones or wait for translation results, and can understand the translated content through the smart watch, thereby improving the efficiency and convenience of communication.

[0063] In an optional embodiment, the method further comprises: An intention of the first user in the communication dialog box is determined, and a strategy for prompting the first user is determined according to the intention of the first user.

[0064] Specifically, determining the intention of the first user in the communication dialog box, and determining a strategy for prompting the first user according to the intention of the first user, includes the following steps: Step 1: Extract all text, voice information, and action semantic information from the communication dialog box. Step 2: Clean the extracted data and remove irrelevant information, such as redundant words and irrelevant characters. Step 3: Use natural language understanding technology, such as intent classification models, to identify the user's intent. Step 4: Based on the identified intent, analyze the user's needs, expectations, and possible confusion points. Step 5: Based on the analysis of user intent, formulate one or more prompt strategies. These strategies include providing additional information, clarifying questions, guiding users to the next step, and recommending relevant options.

[0065] To better illustrate the above steps, here are some possible prompt strategy examples that can be flexibly adjusted according to different user intentions and scenarios: Example 1: Provide additional information strategy. When the information the user asks is not detailed enough or needs to be supplemented. User: "I want to book a hotel." Prompt strategy: "In order to recommend a suitable hotel for you, when is your desired check-in date? And what is your budget range?".

[0066] Example 2: Clarifying question strategy. When the user's question is ambiguous or unclear. User: "How do I use this product?" Prompt strategy: "Do you mean how to install the product or how to use it? If it's how to use it, please tell me which part of the operation you want to know about."

[0067] Example 3: Guide user operation strategy. When the user needs to perform a series of operations but does not know how to do it. User: "I want to change my password." Prompt strategy: "To change your password, please click the 'Account Settings' button, then select the 'Change Password' option, and enter your new password as prompted.".

[0068] Example 4: Recommend related options strategy. When the user is faced with multiple choices but does not know how to choose. User: "I want to find an interest class suitable for my child." Prompt strategy: "Based on your needs, we recommend the following interest classes for you: painting class, dance class and science experiment class. You can choose according to your child's interests and specialties."

[0069] The above prompt strategy can help users complete their tasks more effectively and optimize the user experience. In actual applications, it can be flexibly adjusted and optimized according to the specific needs and scenarios of users.

[0070] This application can accurately identify user intentions through the above steps and formulate corresponding prompt strategies, which can provide users with more personalized and considerate services, and help improve user satisfaction and loyalty. At the same time, the prompt strategy can guide users to perform correct operations, reduce communication barriers caused by insufficient information or misunderstandings, and help speed up communication and interactive experience.

[0071] In an alternative embodiment, see Figure 5 As shown, Figure 5 A flow chart of a method for determining a user prompt strategy provided by an embodiment of the present invention is shown, wherein a strategy for prompting the first user is determined according to the intention of the first user, including steps S501-S502: S501: Determine whether the policy is related to the smart watch.

[0072] Specifically, after formulating the prompt strategy, first check whether the strategy contains information related to the smart watch functions, settings, applications or interactions, and confirm whether the smart watch has the functions and technical capabilities required to display or execute the strategy.

[0073] S502: If the policy is related to the smart watch, send the policy to the smart watch.

[0074] Specifically, if the policy is related to the smart watch and the smart watch has the ability to receive and execute the policy, the policy is sent to the smart watch. After receiving the policy, the smart watch displays it on the screen or notifies the user in other ways (such as voice prompts).

[0075] For example, in the navigation scenario, users can be reminded not only through the information displayed by smart glasses, but also through the vibration of smart watches. For example, when the user is in a downtown area with a lot of people and noisy sounds, it is not appropriate to display the guide route in smart glasses because it will block the user's sight, and it is not appropriate to remind the user through sound because in a noisy environment, the volume of the reminder may cause damage to the user's hearing. Therefore, the navigation strategy is sent to the smart watch at this time, and at the key nodes of navigation, the smart watch reminds the user through vibration, thereby realizing the navigation function.

[0076] This application determines the relevance of the policy and the smart watch through steps S501-S502, ensuring that only policies that match the functions of the smart watch are sent, which can avoid unnecessary waste of resources and improve resource utilization efficiency. The prompts of smart watches are usually more hidden and intuitive, and will not disturb users like traditional mobile phone notifications, which helps to reduce user distraction, improve user concentration and work efficiency, and is suitable for more scenarios.

[0077] In order to better illustrate the information processing method provided by this application, this application also provides a method for translating and responding in real time through smart glasses and watches based on the information processing method. Figure 6 As shown, Figure 6 The flowchart of a method for translating and responding in real time based on smart glasses and watches provided by an embodiment of the present invention is shown, wherein when the wearer of the smart device wears smart glasses and a smart watch at the same time, the user's voice information / gesture information is received through the glasses or the watch, and the voice information of the interlocutor is received through the watch. The smart glasses capture the actions and expressions of the interlocutor through the image sensor, form a communication dialog box on the display interface, and display the translated voice information of the interlocutor, as well as the action information with semantics. Then, the translated voice information / gesture information is expressed to the user through the screen or speaker of the watch. Alternatively, the semantic information in the conversation, the actions and expressions of the interlocutor are recognized, and suggestions are given based on the user's intention; in response to the user executing the suggestion, a reminder strategy is determined based on the user's current state; when the reminder strategy requires the participation of the watch, the reminder strategy is sent to the watch.

[0078] The information processing method provided in this application, by combining the functions of smart glasses and smart watches, optimizes the multimodal interaction experience, information display and dialogue management, cross-device collaboration, personalized prompt strategy and multilingual communication, etc., providing users with a richer, more convenient and safer interactive experience. It can not only accurately and effectively identify the content of user dialogue communication, so that users can accurately know the content of dialogue communication, but also improve the user experience and satisfaction.

[0079] Based on the same application concept, the embodiment of the present invention also provides an information processing device, see Figure 7 As shown, Figure 7 A schematic diagram of the structure of an information processing device provided by an embodiment of the present invention is shown, wherein the device is applied to smart glasses, a first user wears the smart glasses and a smart watch, and the device includes: A first voice information receiving module 701 is used to receive a first voice information sent by a first user; A second voice information receiving module 702 is used to receive a second voice information sent by the smart watch, wherein the voice information is input by a second user to the smart watch; The motion information acquisition module 703 is used to acquire the motion information of the second user through an image sensor; The communication dialog box display module 704 is used to recognize and process the first voice information, the second voice information and the action information, and form a communication dialog box on the display interface of the smart glasses.

[0080] Based on the same application concept, an embodiment of the present application further provides a pair of smart glasses, wherein the smart glasses execute the steps of the information processing method described in any one of the above embodiments when running.

[0081] Based on the same application concept, the present application embodiment also provides a computer device, see Figure 8 As shown, Figure 8 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention is shown, wherein Figure 8 As shown, a computer device 800 provided in an embodiment of the present application includes: A processor 801, a memory 802 and a bus 803, wherein the memory 802 stores machine-readable instructions executable by the processor 801. When the computer device 800 is running, the processor 801 communicates with the memory 802 via the bus 803, and the machine-readable instructions are executed by the processor 801 to execute the steps of the information processing method shown in the above embodiment.

[0082] Based on the same application concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the information processing method described in any one of the above embodiments are executed.

[0083] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0084] The computer program product for performing information processing provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments and will not be repeated here.

[0085] The information processing device provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The system provided in the embodiment of the present invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the system embodiment can refer to the corresponding contents in the aforementioned method embodiment. It can be clearly understood by technicians in the relevant field that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0086] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0087] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0088] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0089] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0090] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0091] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or perform equivalent replacements on some of the technical features thereof; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An information processing method, applied to smart glasses, a first user wears the smart glasses and a smart watch, wherein: The method comprises: Receiving a first voice message sent by a first user; receiving a second voice message sent by the smart watch, wherein the voice message is input to the smart watch by a second user; acquiring the action information of the second user through an image sensor; The first voice information, the second voice information and the action information are recognized and processed, and a communication dialog box is formed on the display interface of the smart glasses.

2. The method according to claim 1, wherein: The first voice information, the second voice information and the action information are recognized and processed, and a communication dialog box is formed on the display interface of the smart glasses, including: The semantics corresponding to the first voice information, the second voice information and the action information are arranged in chronological order and semantic order to form the communication dialog box.

3. The method according to claim 1, wherein: The first voice information, the second voice information and the action information are recognized and processed, and a communication dialog box is formed on the display interface of the smart glasses, including: Determining whether a time interval of the second voice information coincides with a time interval of the action information; In response to the time intervals of the second voice information and the action information not overlapping, the semantic information corresponding to the action information is determined as the content in the communication dialog box.

4. The method according to claim 3, wherein: The first voice information, the second voice information and the action information are recognized and processed, and a communication dialog box is formed on the display interface of the smart glasses, including: In response to the time interval of the second voice information and the action information being coincident, determining an approximate value of the second voice information and the action information; In response to the approximation value being greater than a preset value, the semantic content corresponding to the second voice information is determined as the content in the communication dialog box.

5. The method according to claim 3, wherein: The first voice information, the second voice information and the action information are recognized and processed, and a communication dialog box is formed on the display interface of the smart glasses, including: In response to the time interval of the second voice information and the action information being coincident, determining an approximate value of the second voice information and the action information; In response to the approximation value being less than a preset value, determining a first semantic content corresponding to the second voice information, and determining a second semantic content corresponding to the action information; The first semantic content and the second semantic content are fused, and the fused semantic information is determined as the content in the communication dialog box.

6. The method according to claim 1, wherein: After receiving the first voice message sent by the first user, the method further includes: The first voice information is translated and sent to the smart watch, so that the smart watch displays or plays the translated first voice information.

7. The method according to claim 1, wherein: The method further includes: determining the intention of the first user in the communication dialog box, and determining a strategy for prompting the first user according to the intention of the first user.

8. The method according to claim 7, wherein: Determining a strategy for prompting the first user according to the intention of the first user includes: determining whether the policy is associated with the smart watch; If the policy is related to a smart watch, the policy is sent to the smart watch.

9. An information processing device, applied to smart glasses, a first user wears the smart glasses and a smart watch, wherein: The device comprises: A first voice information receiving module, used to receive a first voice information sent by a first user; A second voice information receiving module, configured to receive a second voice information sent by the smart watch, wherein the voice information is input by a second user to the smart watch; an action information acquisition module, configured to acquire the action information of the second user through an image sensor; The communication dialog box display module is used to identify and process the first voice information, the second voice information and the action information, and form a communication dialog box on the display interface of the smart glasses.

10. A pair of smart glasses, wherein: When the smart glasses are running, the steps of the information processing method as described in any one of claims 1 to 7 are executed.

11. A computer device, wherein: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the information processing method as described in any one of claims 1 to 7 are performed.

12. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the information processing method as claimed in any one of claims 1 to 7 are executed.

Images