A near-eye display device

Abstract

The application discloses a near-eye display device, comprising: a frame body for connecting optical lenses; a wearing part connected with the frame body and used for wearing; a near-eye display module connected with the frame body and / or the optical lenses; a communication module used for being communicatively connected with a terminal device; and a controller communicatively connected with the near-eye display module and the communication module, the controller being configured to: receive a content stream output by the terminal device; acquire auxiliary information, the auxiliary information being information obtained by the terminal device through analysis on at least part of the content in the content stream, determination of a target intelligent agent from intelligent agents according to an analysis result, and processing of the content stream by the target intelligent agent; and control the near-eye display module to display auxiliary information matched with the content stream to a near-eye side according to first attribute information of the content stream and second attribute information of the auxiliary information. The application can flexibly adapt the target intelligent agent based on the content, fully utilize the target intelligent agent to generate the auxiliary information, and effectively improve a conversation assistance effect in a conversation process of a user.

Description

Technical Field

[0001] This application relates to the field of smart wearable device technology, and more particularly to a near-eye display device. Background Technology

[0002] The application of near-eye smart devices is becoming increasingly widespread. For example, AR (Augmented Reality) glasses can provide users with many convenient functions, such as one or more of the following: recognizing the user's eye movement behavior and interacting based on the eye movement behavior, displaying text, displaying images, and playing audio.

[0003] However, in scenarios where users wear near-eye smart devices to converse with others, the conversational assistance provided by near-eye smart devices is not ideal. Summary of the Invention

[0004] This application provides a near-eye display device designed to enhance conversational assistance during user sessions.

[0005] This application provides a near-eye display device including:

[0006] A frame for connecting optical lenses, the frame having an opposing ambient side and a near-eye side;

[0007] The wearing part, connected to the frame, is used to be worn by the user and to project the optical lens into the user's field of vision;

[0008] Near-eye display module, connected to the frame and / or optical lens;

[0009] A communication module, used for communication connection with terminal devices;

[0010] The controller, which communicates with the near-eye display module and the communication module, is used for:

[0011] Receive the content stream output by the terminal device. The content stream is the content obtained by the terminal device from the target data of the acquired target object.

[0012] Acquire auxiliary information, which is information obtained by the terminal device analyzing at least part of the content stream, determining the target intelligent agent from the intelligent agents based on the analysis results of the content stream, and using the target intelligent agent to process the content stream;

[0013] Based on the first attribute information of the content stream and the second attribute information of the auxiliary information, the near-eye display module is controlled to display auxiliary information matching the converted text to the near-eye side.

[0014] In summary, this application provides a near-eye display device, including: a frame for connecting optical lenses, the frame having an ambient side and a near-eye side; a wearing part connected to the frame for wearing on a user and projecting the optical lenses into the user's field of vision; a near-eye display module connected to the frame and / or the optical lenses; a communication module for communicating with a terminal device; and a controller connected to the near-eye display module and the communication module, for: receiving a content stream output by the terminal device, the content stream being content obtained by the terminal device by converting target data of a target object; acquiring auxiliary information, the auxiliary information being information obtained by the terminal device analyzing at least a portion of the content stream, determining a target intelligent agent from intelligent agents based on the analysis results of the content stream, and processing the content stream using the target intelligent agent; and controlling the near-eye display module to display auxiliary information matching the content stream to the near-eye side based on the first attribute information of the content stream and the second attribute information of the auxiliary information. This application can flexibly adapt the target intelligent agent based on the content stream received by the near-eye display device, fully utilize the target intelligent agent to generate auxiliary information, and effectively improve the conversation assistance effect during user conversations.

[0015] Other related beneficial effects of this application will be described in the relevant embodiments. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the usage environment of the near-eye display device provided in the embodiments of this application;

[0018] Figure 2 This is a schematic diagram illustrating information interaction between a near-eye display device and a terminal device, provided in an embodiment of this application.

[0019] Figure 3 This is another schematic diagram of information interaction between a near-eye display device and a terminal device provided in an embodiment of this application;

[0020] Figure 4 This is another schematic diagram illustrating information interaction between a near-eye display device and a terminal device, provided in an embodiment of this application.

[0021] Figure 5 This is a schematic diagram illustrating another information interaction between a near-eye display device and a terminal device provided in an embodiment of this application.

[0022] Figure 6This is a schematic diagram of an intelligent agent in a terminal device provided in an embodiment of this application;

[0023] Figure 7 This is a schematic diagram of a terminal device filtering target intelligent agents provided in an embodiment of this application;

[0024] Figure 8 A schematic block diagram of a near-eye display device provided in an embodiment of this application;

[0025] Figure 9 This is a schematic diagram of a display interface in a near-eye display device provided in an embodiment of this application;

[0026] Figure 10 A schematic diagram of a near-eye display device provided in an embodiment of this application;

[0027] Figure 11 An exploded view of a near-eye display module in a near-eye display device provided in this application embodiment;

[0028] Figure 12 This is a schematic diagram of another display interface in a near-eye display device provided in an embodiment of this application;

[0029] Figure 13 A schematic diagram of yet another display interface in a near-eye display device provided in an embodiment of this application;

[0030] Figure 14 This is a schematic diagram illustrating the active interaction of auxiliary information between different intelligent agents in a chat scenario provided in an embodiment of this application.

[0031] Figure label:

[0032] 100. Near-eye display device; 10. Frame; 20. Wearing part; 21. Temple; 30. Near-eye display module; 31. Microdisplay; 32. Optical module; 321. External thread; 33. Display interface; 34. First area; 35. Second area; 36. Front cover; 361. First through hole; 362. Internal thread; 37. Rear cover; 38. Flexible sleeve; 381. Second through hole; 39. Receiving cavity; 40. Communication module; 50. Controller; 61. Environmental side; 62. Near-eye side; 70. Data acquisition module; 80. Optical lens;

[0033] 200. Terminal device; 300. First terminal device; 400. Second terminal device; 500. Collection of intelligent agents; 501. First type of intelligent agent; 502. Second type of intelligent agent; 503. Third type of intelligent agent; 504. Target intelligent agent. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, combined, or partially merged; therefore, the actual execution order may change according to the actual situation.

[0035] Please refer to the following first. Figure 1 , Figure 1 This is a schematic diagram of the usage environment of the near-eye display device 100 provided in this application embodiment. Taking an environment including the near-eye display device 100 and at least one terminal device 200 as an example, the near-eye display device 100 and at least one terminal device 200 form a communication connection, and the near-eye display device 100 can receive content streams output by the terminal devices 200. These content streams include, but are not limited to, voice, text, video, etc. As one embodiment, the near-eye display device 100 is provided with a communication module 40, through which the near-eye display module can communicate with at least one terminal device 200. In some embodiments, the terminal device 200 may also include BLOOMZ (autoregressive model), GPT (Generative Pre-Trained Transformer), ChatGLM (General Language Model), LLaMA (Large Language Model MetaAI), OPT (Open Pretrained Transformer), Claude model, Qwen, deepseek, or Wenxin Yiyan, etc., as well as other general or specialized large models.

[0036] For example, the terminal device 200 includes, but is not limited to, at least one of electronic devices, external cloud computing platforms, external servers, and local servers, wherein electronic devices are, for example, personal computers, mobile phones, smartphones, smartwatches, smart bracelets, smart rings, etc.

[0037] Specifically, the terminal device 200 is configured to convert the target data of the acquired target object into a content stream. The target data can be data from conversations between the wearer and an object other than the wearer (which can be called the interaction object), and its form includes, but is not limited to, voice, text, video, etc.

[0038] like Figures 2 to 3As shown, in one embodiment, the target data is acquired by the near-eye display device 100. For example, it is acquired through a data acquisition module 70 disposed in the near-eye display device 100. The data acquisition module 70 includes, but is not limited to, a voice data acquisition module, an image data acquisition module, a text data acquisition module, and a motion data acquisition module. For example, the voice data acquisition module may be a single or arrayed microphone; the image data acquisition module may be an optical sensor, a digital camera, or a CCD camera; the text data acquisition module may be a keyboard; and the motion data acquisition module may be at least one of a position sensor, an inertial sensor, or a biometric sensor.

[0039] Specifically, the near-eye display device 100 can transmit the collected target data to the terminal device 200, so that the terminal device 200 can convert the target data into a content stream, and then the terminal device 200 can transmit the content stream back to the near-eye display device 100.

[0040] like Figure 4 As shown, in another embodiment, the target data is acquired by the terminal device 200, for example, through an acquisition unit located on the terminal device 200. The acquisition unit may include a voice data acquisition module, an image data acquisition module, a text data acquisition module, and a motion data acquisition module. For example, the voice data acquisition module may be a single or arrayed microphone; the image data acquisition module may be an optical sensor, a digital camera, or a CCD camera; the text data acquisition module may be a keyboard; and the motion data acquisition module may be at least one of a position sensor, an inertial sensor, or a biometric sensor. Specifically, after the terminal device 200 acquires the target data, it converts the target data into a content stream and then transmits the content stream to the near-eye display device 100.

[0041] Specifically, the process by which the terminal device 200 and / or the near-eye display device 100 converts the target data to obtain a content stream includes, but is not limited to, converting the target data into text, images, videos, or instructions corresponding to the near-eye display device 100. The conversion of the target data can be implemented using the relevant large models listed in the above embodiments, or it can be obtained using deep learning based on convolutional neural networks. As one embodiment, the target data includes voice data during conversational communication between the wearer and the interactive object, and the conversion of the target data includes converting the voice data into text data as a content stream.

[0042] The process of converting voice data into text data can utilize a large language model (such as LLM), an automatic speech recognition model, a speech-to-text model, a text recognition and conversion model, etc., preset in the terminal device 200 and / or near-eye display device 100.

[0043] like Figures 6 to 7 As shown, the terminal device 200 is further configured to analyze at least a portion of the content stream, determine a target agent 504 from the agent set 500 based on the analysis results, and use the target agent 504 to process the content stream to obtain auxiliary information. The agent set 500 includes multiple agents (e.g., a first type of agent 501, a second type of agent 502, and a third type of agent 503 as described below). The terminal device 200 can filter and / or extract key content from the content stream, and then analyze the processed portion of the content. In some embodiments, the agent can be built from a basic model of an LLM (Large Language Model); the agent can also be built from an existing large model, or by fine-tuning an existing large model based on training samples; in some embodiments, the agent also includes multiple agents with different modes, each built from a different large model, for example, one agent is built using a general large model, and another agent is built and fine-tuned using local user data.

[0044] The terminal device 200 may be pre-loaded with at least two selectable intelligent agents. The target intelligent agent 504 refers to at least one intelligent agent that matches the content stream, and different intelligent agents are configured to perform different processing on the content stream to generate corresponding auxiliary information.

[0045] In some implementations, auxiliary information includes, but is not limited to, at least one of the following: detection results obtained from fact detection of the content stream; recommended response information generated based on the content stream; key words and phrases extracted from the content stream and reference background information generated based on these key words and phrases; translated text obtained by translating the content stream; professional prompts given in combination with the content stream and the current context, such as "Please show enthusiasm and confidence, and be prepared to answer questions" when referring to someone else's question; or analyzing emotional parameters in the content stream, such as judging that the emotion is intense if there is an intensified tone. The auxiliary information is generated by one of the aforementioned agents based on the content stream. In some embodiments, one agent can be configured as a general or generic agent. A general agent generates auxiliary information in real-time based on all conversation content streams. For example, if the content stream contains communication information about "it rained heavily in Shenzhen yesterday," the general agent performs fact-checking and then outputs auxiliary information such as "it rained moderately in Shenzhen yesterday, and the wind was level xx." Another agent is configured as an active or special agent, configured to provide users with real-time conversation content strategies based on the current context. Similarly, if the content stream contains communication information about "it rained heavily in Shenzhen yesterday," the active agent will also... The weather information for the day is verified. At this point, the proactive agent outputs auxiliary information such as, "Please express your doubts and provide feedback based on facts, for example: My colleague was in Shenzhen yesterday, and I heard it was sunny there." It can be understood that one agent is configured to generate auxiliary information feedback based on the content stream itself, while the other agent is configured to generate auxiliary information feedback based on the content stream and the context it creates. This context includes the relationships, knowledge backgrounds, and familiarity among the parties involved in the conversation. The proactive agent generates professional or skill-based auxiliary information based on the conversational context, which can improve communication efficiency and mutual goodwill.

[0046] like Figure 5 As shown, in some embodiments, the usage environment of the near-eye display device 100 includes at least two different terminal devices, a first terminal device 300 and a second terminal device 400. As one embodiment, the first terminal device 300 is configured to convert target data to obtain a content stream, and the second terminal device 400 is configured to analyze at least a portion of the content stream, determine a target intelligent agent 504 from the intelligent agent set 500 based on the analysis results of the content stream, and use the target intelligent agent 504 to process the content stream to obtain auxiliary information.

[0047] In some embodiments, combined with Figure 5The first terminal device 300 acquires target data and converts it to obtain a content stream. It can simultaneously send the content stream to the near-eye display device 100 and the second terminal device 400. The near-eye display device 100 and the first terminal device 300 can simultaneously display the content stream. The second terminal device 400 generates auxiliary information based on the content stream and sends the auxiliary information to the first terminal device 300 for display. The first terminal device 300 then sends the auxiliary information to the near-eye display device 100 for display. Alternatively, the second terminal device 400 can directly send the auxiliary information to the near-eye display device 100. In other embodiments, the near-eye display device 100 can acquire the content stream and (i.e., the dashed box portion 1010) display the content stream, and then send the displayed content stream to the first terminal device 300 and the second terminal device 400 (not shown in the figure).

[0048] In other embodiments, after the first terminal device 300 generates a content stream, it transmits the content stream to the near-eye display device 100. The near-eye display device 100 then transmits the content stream to the second terminal device 400. The second terminal device 400 generates auxiliary information based on the content stream and transmits the auxiliary information to the near-eye display device 100. Alternatively, after the first terminal device 300 generates a content stream, it directly transmits the content stream to the second terminal device 400. The second terminal device 400 generates auxiliary information based on the content stream and transmits the auxiliary information to the near-eye display device 100.

[0049] For example, the first terminal device 300 is an electronic device carried by the user, such as a smartphone, laptop, tablet or desktop computer, smart bracelet, smart ring, etc., and the second terminal device 400 is at least one of an external cloud computing platform, an external server, and a local server. It should be understood that the computing power configuration of the second terminal device 400 is greater than that of the first terminal device 300, and it can support the operation of the intelligent agent.

[0050] Please see Figure 8 and Figure 9 This application provides a near-eye display device 100, which may be smart glasses, a smart helmet, or other wearable devices on a user's head.

[0051] like Figure 8 As shown, the near-eye display device 100 includes at least a frame 10, a wearable part 20, a near-eye display module 30, a communication module 40, and a controller 50. The functions and structures of each module are described in detail below.

[0052] Specifically, the frame 10 is used to connect the optical lens 80, and the frame 10 has an opposing ambient side 61 and a near-eye side 62. It should be understood that when the optical lens 80 is connected to the frame 10, the optical lens 80 is used to adjust the light emitted from the external environment to the user's eye. In some embodiments, the frame 10 may not be connected to the optical lens 80. For example, the optical lens 80 may be a plano lens, a refractive lens, or a superlens.

[0053] Specifically, the wearing part 20 is connected to the frame 10 and is used to wear the device on the user and project the optical lens 80 in front of the user's field of vision. Specifically, when the user wears the near-eye display device 100 on their head through the wearing part 20, the optical lens 80 is located in front of the user's eyes.

[0054] For example, the wearing part 20 can be at least one of the temple 21 or the helmet strap. For example, when the near-eye display device 100 is smart glasses, the wearing part 20 is the temple 21, or when the near-eye display device 100 is a smart helmet, the wearing part 20 is the helmet strap.

[0055] Specifically, the near-eye display module 30 is connected to the frame 10 and / or the optical lens 80 and is configured to display information to the near-eye side 62, including but not limited to at least one of text, images, and logos, so that the user can obtain the displayed information. As one embodiment, the near-eye display module includes a display interface 33 opposite to the near-eye side 62, and the near-eye display module is configured to display information on the display interface 33, thereby achieving the concealed presentation of the displayed information.

[0056] like Figure 10 As shown, in some embodiments, the near-eye display module 30 is disposed on the frame 10, for example, located on the upper edge of the frame 10. As an example, the near-eye display module 30 is detachable from the frame 10 and / or can be moved relative to it. For example, the user can adjust the near-eye display module 30 according to their own visual pupil distance range, such as moving it horizontally left and right, rotating it up and down, etc.

[0057] As one embodiment, the near-eye display module 30 includes a microdisplay 31 and an optical module 32. The microdisplay 31 is disposed on the frame 10 and configured to output image light to the near-eye side 62. The optical module 32 is disposed on the near-eye side 62 of the microdisplay 31 to form a display interface 33 on the near-eye side 62 of the optical module 32. The optical module 32 is used to perform preset processing on the image light output by the microdisplay 31. Exemplarily, the optical module 32 is at least used to perform divergence processing on the image light. Exemplarily, the optical module 32 can be adjusted in distance relative to the microdisplay 31 to achieve adjustable focus, thereby adjusting the size of text, images, and logos displayed on the display interface 33.

[0058] like Figure 11As shown, in one embodiment, the near-eye display module 30 may include a front cover 36, a rear cover 37, and a flexible sleeve 38. The wearing part 20 includes a temple 21. The microdisplay 31 is located within the accommodating cavity 39 formed by the front cover 36 and the rear cover 37. The temple 21 contains components such as a battery and a circuit board (not shown). The microdisplay 31 is electrically connected to the battery or circuit board and other electrical components of the temple 21 via the flexible circuit board. The optical module 32 is located on the light-emitting side of the microdisplay 31. The outer periphery of the optical module 32 is provided with an external thread 321. The front cover 36 is provided with a first through hole 361. The inner circumferential wall of the first through hole 361 is provided with an internal thread 362, and the flexible sleeve 38 is provided with a second through hole 381. The flexible sleeve 38 is located between the optical module 32 and the front cover 36. The external thread 321 and the internal thread 362 are threaded together to compress and deform the flexible sleeve 38, thereby effectively fixing the optical module 32 to the light-emitting side of the microdisplay 31. The light from the microdisplay 31 can be projected to the user's eye through the optical module 32 toward the near-eye side 62. For example, the light from the microdisplay 31 is projected to the user's eye after multiple reflections inside the optical module 32. The microdisplay 31 may include Micro-LED (Micro Light-Emitting Diode), uLED (Micro Light-Emitting Diode), Micro-oled (Micro Organic Light-Emitting Diode), LCoS (Liquid Crystal on Silicon), LCD (Liquid Crystal Display), DMD (Digital Micromirror Device) / DLP (Digital Light Processing) or LBS (Laser Beam Scanning), or any combination thereof. The optical module 1108 may be cylindrical, and its projected area along the optical axis may be elliptical, circular, or polygonal. Therefore, the outline of the display interface 33 may be circular, elliptical, or polygonal. The optical module 1108 may also adopt optical solutions such as CN202322496484.0 and CN202410306460.0. In other embodiments, the light-emitting side of the optical module 30 may also be provided with other optical lenses (not shown) adapted for correcting the user's visual impairment, such as myopia lenses, hyperopia lenses, etc.

[0059] like Figure 9As shown, in some other embodiments, the near-eye display module 30 can also be connected to the optical lens 80, that is, the image of the display module 30 is directly displayed on the optical lens 80. For example, the near-eye display module 30 includes a waveguide lens and a projection component. The waveguide lens is connected to and stacked with the optical lens 80. The projection component is set corresponding to the coupling area of ​​the waveguide lens and is configured to output image light to the coupling area, so as to display a projected image corresponding to the image light in the coupling area of ​​the waveguide lens using the waveguide principle of the waveguide lens. At this time, the coupling area of ​​the waveguide lens forms the display interface 33. In some other embodiments, the near-eye display module 30 is connected to both the frame 10 and the optical lens 80.

[0060] Specifically, the communication module 40 is used to communicate with the terminal device 200 to support communication between the near-eye display device 100 and the terminal device 200, including but not limited to the communication transmission of at least one of the target data, content stream and auxiliary information mentioned above. For example, the communication module 40 and the terminal device 200 can communicate via wired or wireless communication. The form of wireless communication connection between the communication module 40 and the terminal device 200 includes one or a combination of the following mobile communication (GSM) network, GPRS network, enhanced data GSM environment (EDGE) network, IEEE 802.xx communication network (e.g., WiFi, WiMax, ZigBeeTM), 3G, 4G, 4G LTE, 5G, code division multiple access (CDMA) network, near field communication (NFC), WiFi Direct, infrared and Bluetooth networks, etc.

[0061] Specifically, the controller 50 is communicatively connected to the near-eye display module 30 and the communication module 40, and is used to: receive the content stream output by the terminal device 200, the content stream being the content obtained by the terminal device 200 by converting the target data of the acquired target object;

[0062] The auxiliary information is obtained by the terminal device 200 analyzing at least part of the content in the content stream, determining the target intelligent agent 504 from the intelligent agent set 500 based on the analysis results of the content stream, and using the target intelligent agent 504 to process the content stream.

[0063] Based on the first attribute information of the content stream and the second attribute information of the auxiliary information, the near-eye display module 30 is controlled to display auxiliary information matching the content stream to the near-eye side 62.

[0064] It should be noted that the first attribute information of the content stream is used to characterize the attributes of the content stream, including but not limited to the time node, content type, and source information (including the source object and the source agent). The first attribute information and the second attribute information are also used to characterize the matching relationship between the content stream and auxiliary information, such as characterizing which part of the content stream a certain auxiliary information is associated with.

[0065] Based on this, the controller 50 can determine the matching relationship between the content stream and the auxiliary information according to the first attribute information and the second attribute information, and control the near-eye display module 30 to display the auxiliary information that matches the content stream. It should be understood that the near-eye display module 30 can display the matching content stream and auxiliary information simultaneously, sequentially, or only the auxiliary information.

[0066] As described above regarding the terminal device 200, the terminal device 200 is pre-loaded with an agent set 500 containing at least two selectable agents. The target agent 504 refers to at least one agent 501, etc., that matches the content stream. Different agents in the agent set 500 are configured to perform different processing on the content stream to generate corresponding auxiliary information. The terminal device 200 is configured to analyze at least a portion of the content stream, determine the target agent 504 from the agents based on the analysis results, and use the information obtained by processing the content stream using the target agent 504.

[0067] It should be understood that in a conversational scenario between a user (wearing object) and an interaction object, different content streams can be adapted to different types of auxiliary information. For example, some content streams may include declarative sentences appearing in the conversation, in which case fact detection is required to obtain the detection results; or, some content streams may include statements corresponding to the interaction object, in which case recommended response information corresponding to the wearing object needs to be generated; or, some content streams may include professional terms or high-frequency words and phrases appearing in the conversation, in which case these words and phrases need to be extracted from the content stream as key words and phrases, and reference background information generated based on the key words and phrases; or, some content streams may include foreign language statements appearing in the conversation (it is easy to know that foreign languages ​​here refer to languages ​​that the user does not know), in which case translation processing is required to obtain translated text; if some content streams include interrogative sentences, then these interrogative sentences need to be answered; the above process can be that the content stream input is transformed by different intelligent agents and then output to obtain different auxiliary information.

[0068] In some embodiments, the intelligent agent may include a general intelligent agent (or a general intelligent agent) and an active intelligent agent (or a special intelligent agent). The general intelligent agent provides transformation feedback for each sentence of the conversation, such as: whether the facts are accurate, investigation of the domain mentioned in the topic, retrieval of the knowledge graph, knowledge verification, etc. The active intelligent agent provides deeper active feedback based on the content of the conversation, used to infer the intention or purpose of the conversation content, such as additional data mining based on the conversation content combined with the context, professional tips, sentiment tips, and skill tips. Users can select and adjust the transformation and appearance order and priority of different intelligent agents according to the needs of the conversation scenario. For example, users can choose to have both a general intelligent agent and an active intelligent agent participate in the conversation process at the same time, where the feedback priority of the general intelligent agent is higher than that of the active intelligent agent, that is, the auxiliary information corresponding to the general intelligent agent is presented before the auxiliary information corresponding to the active intelligent agent; or the feedback priority of the general intelligent agent is lower than that of the active intelligent agent, that is, the auxiliary information corresponding to the active intelligent agent is presented before the auxiliary information corresponding to the general intelligent agent. In one scenario, for example, during a conversation between Person A (wearing a near-eye display device) and Person B (not wearing a near-eye display device), the content stream might include: Person B: "I heard that Company XX has released a new large model that can predict the weather." The general agent performs fact detection and outputs auxiliary information such as "Company XX released large model N1 yesterday, whose main function is future weather prediction, specifically including wind speed, temperature, etc." The proactive agent, recognizing the conversation's intent (or orientation) as wanting to know about potential business opportunities, infers the topics of interest for both parties regarding the weather prediction model based on the context, performs deeper proactive feedback, and provides auxiliary information such as "Focus on the business value and potential opportunities of the weather prediction model." Simultaneously, the proactive agent collects relevant data or industry reports, and then further generates auxiliary information such as "Related reports and data on the business opportunities related to weather prediction by the large model," which is then displayed to Person A. Of course, Person A can choose to present either the general agent or the proactive agent with higher priority. In some embodiments, the user can also choose to display any one or a combination of auxiliary information converted from an active agent and auxiliary information converted from a general agent on the display interface 33 (described below) of the near-eye display module 100, wherein the auxiliary information output by the agent can be automatically switched and presented according to the content of the conversation. Therefore, the user can communicate more efficiently based on the auxiliary information generated by the active agent, thereby effectively improving communication efficiency and effectively bringing the relationship and goodwill between the two parties in the current context.

[0069] In some embodiments, the first terminal device 300 and the second terminal device 400 simultaneously store auxiliary information for active agent conversion and auxiliary information for general agent conversion, meaning that data can still be traced after power-on and power-off. However, the near-eye display device 100 does not store auxiliary information for active agent conversion or general agent conversion, meaning that data cannot be traced after power-on and power-off. In other embodiments, the near-eye display device 100 simultaneously stores auxiliary information for active agent conversion and general agent conversion, thereby effectively reducing the possibility of data loss.

[0070] In some embodiments, the agents in the agent set 500 are configured to detect the content stream during the conversation in real time and provide proactive auxiliary information feedback. This agent does not require the user to actively initiate the conversation or input content into an input box, nor does it require triggering specific wake-up words, such as saying "Hey Siri" during the conversation. The agent set 500 includes at least one proactive agent capable of providing proactive auxiliary feedback for every sentence that appears during the conversation. The content stream of the conversation can originate from the wearer of the near-eye display device, or from the wearer's surrounding environment or the person being communicated with. In some embodiments, the number of proactive agents can also include multiple agents. Each proactive agent can be assigned to predict conversation content intents in different domains. For example, some proactive agents are used to proactively infer auxiliary information related to health intents in the conversation content, some are used to proactively infer auxiliary information related to education intents in the conversation content, some are used to proactively infer auxiliary information related to financial intents in the conversation content, and so on.

[0071] Based on this, the near-eye display device 100 provided in this application embodiment can flexibly adapt to the target intelligent agent 504 based on the content stream, so as to make full use of the target intelligent agent 504 to generate auxiliary information, so that the obtained auxiliary information meets the actual auxiliary needs of the user in the conversation scenario, and effectively improves the auxiliary effect of the near-eye display device 100 on the user's conversation process.

[0072] In some embodiments, the near-eye display module includes a display interface 33 opposite to the near-eye side 62, and the controller 50 is further configured to:

[0073] The association between content stream and auxiliary information is determined based on the first attribute information and the second attribute information;

[0074] Based on the association relationship, the near-eye display module will associate and display the content stream with the auxiliary information associated with the content stream on the display interface 33.

[0075] Specifically, the first attribute information and the second attribute information are also used to characterize the matching relationship between the content stream and the auxiliary information, such as characterizing which part of the content stream a certain auxiliary information is associated with; the controller 50 controls the near-eye display module to associate and display the content stream and the auxiliary information associated with the content stream on the display interface 33 based on the above association relationship.

[0076] It should be noted that if the terminal device 200 generates auxiliary information based on a certain sub-content in the content stream, then the auxiliary information is associated with the sub-content and has matching attribute information. The controller 50 can then determine the association between the content stream and the auxiliary information based on the first attribute information and the second attribute information.

[0077] like Figure 12 As one embodiment, the near-eye display module can immediately display the content stream on the display interface 33 after receiving it, and then, after receiving auxiliary information, associate the auxiliary information with the corresponding content stream for display. For example, it can simultaneously or sequentially display the content stream and the associated auxiliary information in the same area of ​​the display interface 33. Figure 13 Alternatively, after receiving the content stream and associated auxiliary information, both can be displayed on the same screen on the display interface 33; or, after determining that there is no matching auxiliary information for the content stream, the auxiliary information can be omitted, and the content stream can be displayed separately on the display interface 33.

[0078] It should be understood that associating related content streams with auxiliary information can intuitively reflect what specific content the auxiliary information is assisting, effectively improving the assistive effect of the near-eye display device 100 on the user's conversation process.

[0079] In some implementations, the converted text and the auxiliary information associated with the converted text have the same text attributes on the display interface 33, including but not limited to at least one of text color, text font, and text size.

[0080] It should be noted that the converted text in the content stream can be obtained by converting at least one of the target data, namely, speech data, text data, video data, and motion data. Specifically, the conversion process can include at least one or a combination of the following: performing speech-to-text conversion on speech data to obtain converted text; performing at least one of the following on text data, namely, translation processing, key point extraction, and invalid information filtering to obtain converted text; extracting audio track data from video data and performing speech-to-text conversion on the audio track data to obtain converted text; generating descriptive text based on video data / motion data as converted text.

[0081] It should be understood that the associated conversion text and auxiliary information have the same text attributes, which makes it easier for users to quickly determine the auxiliary information corresponding to the content stream when viewing the display interface 33, thereby improving the conversational assistance effect.

[0082] For example in Figure 12 In the text, auxiliary information a is auxiliary information generated by terminal device 200 based on converted text A. Therefore, converted text A is associated with auxiliary information a, and the two have the same text font, the same text size, and are both green on the display interface 33.

[0083] In some implementations, the content stream and associated auxiliary information are displayed in the same area of ​​the display interface 33. It should be understood that displaying the associated conversion text and auxiliary information in the same area makes it easier for users to quickly identify the conversion text corresponding to the auxiliary information when viewing the display interface 33, thus improving the conversation assistance effect.

[0084] For example in Figure 12 In this example, auxiliary information 'a' is generated by terminal device 200 based on converted text A. Converted text A is associated with auxiliary information 'a', and both are displayed in the first area 34 of the display interface 33. Auxiliary information 'b' is generated by terminal device 200 based on converted text B. Converted text B is associated with auxiliary information 'b', and both are displayed in the first area 34 of the display interface 33. In one embodiment, converted text A is obtained by converting the conversational voice of the wearable object, and the first area 34 is located on the right side of the display interface 33. In another embodiment, converted text B is obtained by converting the conversational voice of the wearable object's interactive object, and the second area 35 is located on the left side of the display interface 33.

[0085] In some implementations, the controller 50 is also used to:

[0086] The source information of the auxiliary information is determined based on the attribute information of the auxiliary information. The source information includes at least one of the source object corresponding to the auxiliary information and the target intelligent agent 504, wherein the content stream output by the source object corresponds to the auxiliary information.

[0087] The near-eye display module is controlled to associate and display auxiliary information with the same source information.

[0088] It should be noted that "same source information" means that the source object is the same, the corresponding target intelligent agent 504 is the same, or both the source object and the corresponding target intelligent agent 504 are the same. For example, the source object is the wearer of this near-eye display device 100 or an interactive object that communicates with the wearer. The terminal device 200 generates auxiliary information based on the content stream output by the source object. It is easy to know that the target intelligent agent 504 refers to the intelligent agent that processes the content stream.

[0089] Specifically, after determining the source information, the near-eye display module associates and displays auxiliary information with the same source information. The specific association display format can be referred to the description of the aforementioned embodiment.

[0090] For example in Figure 13 In the case of auxiliary information a and auxiliary information d, which are auxiliary information generated by the same intelligent agent mounted on terminal device 200, auxiliary information a and auxiliary information d are related. The two have the same text font, the same text size, and are both green on the display interface 33.

[0091] For example in Figure 13 In the process, if the converted text A and the converted text C are obtained from the conversational speech of the same object (wearable object or interactive object), then the converted text A and the converted text C are associated, and both are displayed in the first area 34 on the display interface 33.

[0092] like Figure 12 and Figure 13 As one embodiment, displaying auxiliary information with the same source information in association also includes: displaying auxiliary information and object icons corresponding to the source information in association, such as displaying the corresponding object icon on one side of the auxiliary information, wherein the object icon includes, but is not limited to, the avatar icon of the source object and the avatar icon of the target intelligent agent 504.

[0093] It should be understood that the above-mentioned associated display method makes it easier for users to understand the source attributes of the assistance information. In particular, since different agents are configured to process the content stream differently, associating the assistance information with the avatar icon corresponding to the target agent 504 can clearly show which agent provided assistance, and / or what kind of assistance the agent provided, effectively improving the conversation assistance effect.

[0094] In some implementations, the auxiliary information includes at least two display messages corresponding to information from different sources, and the controller 50 is further configured to display the display messages with different sources in different areas on the display interface 33. It should be understood that the above display method helps users distinguish between display messages from different sources.

[0095] As one embodiment, the controller 50 is used for;

[0096] Based on the source object, determine the first display information corresponding to the first source information and the second display information corresponding to the second source information from the auxiliary information, wherein the first source information and the second source information are different source information;

[0097] The near-eye display module is controlled to display first display information in the first area 34 and second display information in the second area 35;

[0098] Among them, the first area 34 and the second area 35 are different areas on the display interface 33.

[0099] For example Figure 12 As shown, auxiliary information a is the auxiliary information generated by the terminal device 200 based on the converted text A, i.e., the first display information; auxiliary information b is the auxiliary information generated by the terminal device 200 based on the converted text B, i.e., the second display information. The near-eye display module displays auxiliary information a in the first area 34 and auxiliary information b in the first area 34. As one embodiment, the first area 34 is located on the right side of the display interface 33, and the second area 35 is located on the left side of the display interface 33.

[0100] It should also be noted that in some embodiments, the first region 34 and the second region 35 do not overlap at all, but in other embodiments, the first region 34 and the second region 35 may partially overlap in order to make full use of the area of ​​the display interface 33.

[0101] Furthermore, when there is other information in the auxiliary information that has a different source from the first display information and the second display information, the controller 50 is also used to: determine the third display information corresponding to the third source information from the auxiliary information, and control the near-eye display module to display the third display information in the third area, wherein the third area is a different area from the first area 34 and the second area 35.

[0102] In some implementations, the controller 50 is also used to:

[0103] For example Figure 12 , 13 As shown, the first object icon corresponding to the first source information is displayed in the first area 34; and the second object icon corresponding to the second source information is displayed in the second area 35. It should be understood that the above-described associated display method clearly reflects the source information of the displayed information, effectively improving the conversation assistance effect.

[0104] For example, the first object icon and the second object icon may be avatar icons that match the source object and / or the target agent 504. For example, the first object icon may be displayed on the side of the first display information, and the second object icon may be displayed on the side of the second display information.

[0105] It should be noted that, in addition to displaying auxiliary information with the same source information together, it is also possible to display converted text with the same source information together. In some of the embodiments provided below, converted text with the same source information have the same text attributes on the display interface 33. The text attributes include, but are not limited to, at least one of text color, text font, and text size.

[0106] It should be noted that converted text with the same source information refers to text converted from the content stream output by the same object (wearable object or interactive object). It should be understood that the aforementioned associated display facilitates a clear indication of the source of auxiliary information, providing a direct visual understanding of the conversation content output by the wearable object and the interactive object respectively.

[0107] For example in Figure 13 In the process, the converted text A and the converted text C are obtained by converting the conversational speech of the wearable object. Therefore, the converted text A and the converted text C have the same font and the same text size on the display interface 33, and both are black.

[0108] In some implementations, the controller 50 is also used to:

[0109] When it is determined that there is no matching auxiliary information in the content stream, the near-eye display module is controlled not to display auxiliary information corresponding to the converted text.

[0110] For example in Figure 13 If the converted text C has no matching auxiliary information, the near-eye display module will only display the converted text C on the display interface 33, without displaying any auxiliary information.

[0111] It should be understood that when terminal device 200 lacks corresponding auxiliary information based on a portion of the content stream, terminal device 200 may choose not to transmit auxiliary information to near-eye display device 100, and / or transmit a signal indicating the absence of auxiliary information to near-eye display device 100. For example, when a portion of the content stream does not meet the triggering conditions for the agent to output auxiliary information, and / or when inputting that portion of the content into the agent yields no corresponding output, terminal device 200 may determine that the converted text lacks matching auxiliary information.

[0112] like Figures 2 to 5 As shown, in some embodiments, the controller 50 is also used for:

[0113] The content stream is output to the terminal device 200 so that the terminal device 200 generates matching auxiliary information based on the content stream;

[0114] Receive auxiliary information output by terminal device 200.

[0115] Specifically, as described above regarding the usage environment of the near-eye display device 100, after receiving the content stream, the controller 50 transmits the content stream to the terminal device 200, which then generates auxiliary information. It should be understood that the terminal device 200 is configured to analyze the content stream, determine the target agent 504 from the agents based on the analysis results, and process the content stream using the target agent 504. The above method and steps place high demands on the device's configuration. Therefore, having the terminal device 200 generate the auxiliary information reduces the configuration requirements for the near-eye display device 100, and also indirectly reduces the size and weight of the controller 50 and the power consumption of the near-eye display device 100.

[0116] It should be noted that the conversion and generation of target data and the generation of auxiliary information based on the content stream can be performed by the same terminal device 200 or by different terminal devices 200. For example, in some of the following embodiments, the controller 50 is also used for:

[0117] Receive the content stream output by the first terminal device 300;

[0118] The content stream is output to the second terminal device 400 so that the second terminal device 400 generates matching auxiliary information based on the content stream;

[0119] Receive auxiliary information output by the second terminal device 400;

[0120] Among them, the first terminal device 300 and the second terminal device 400 are different terminal devices 200.

[0121] Specifically, as described above regarding the usage environment of the near-eye display device 100, the content stream is generated by the first terminal device 300 by converting the target data, and the auxiliary information is generated by the second terminal device 400 based on the content stream. The first terminal device 300 can either receive and convert the target data collected by the near-eye display device 100, or it can convert the target data collected by the first terminal device 300 itself.

[0122] As one embodiment, the first terminal device 300 is an electronic device carried by the user, such as a smartphone, laptop, tablet or desktop computer, smart bracelet, smart ring, etc., and the second terminal device 400 is at least one of an external cloud computing platform, an external server, and a local server. It should be understood that the computing power configuration of the first terminal device 300 and the second terminal device 400 is greater than that of the first terminal device 300, and can support the operation of the intelligent agent.

[0123] like Figure 8As shown, in some embodiments, the near-eye display device 100 further includes a data acquisition module 70 that is communicatively connected to the controller 50. The data acquisition module 70 is configured to perform data acquisition, and the data acquired by the data acquisition module 70 includes at least one of voice, text, and video.

[0124] Controller 50 is also used for:

[0125] The control data acquisition module 70 acquires target data of the target object and receives the target data;

[0126] The target data is transmitted to the terminal device 200 via the communication module 40, so that the terminal device 200 can convert the target data to obtain a content stream;

[0127] Receive the content stream output by terminal device 200.

[0128] Specifically, the near-eye display device 100 acquires target data through the data acquisition module 70. As one embodiment, the data acquisition module 70 includes, but is not limited to, a voice data acquisition module, an image data acquisition module, a text data acquisition module, and a motion data acquisition module. For example, the voice data acquisition module may be a single or arrayed microphone; the image data acquisition module may be an optical sensor, a digital camera, or a CCD camera; the text data acquisition module may be a keyboard; and the motion data acquisition module may be at least one of a position sensor, an inertial sensor, or a biometric sensor.

[0129] like Figures 6 to 7 As shown, in some embodiments, the intelligent agent includes at least a first type of intelligent agent 501 and a second type of intelligent agent 502 that is different from the first type of intelligent agent 501;

[0130] The first type of intelligent agent 501 is configured to perform: fact detection on the content stream to obtain the detection result; generate recommended response information corresponding to the wearer of the near-eye display device 100 based on the content stream; extract key words and phrases from the content stream and generate corresponding reference background information based on the key words and phrases; and perform translation processing on the content stream to obtain at least one of the corresponding translated texts.

[0131] The second type of intelligent agent 502 is configured to perform: fact detection on the content stream to obtain the detection result; generate recommended response information corresponding to the wearer of the near-eye display device 100 based on the content stream; extract key words and phrases from the content stream and generate corresponding reference background information based on the key words and phrases; and perform translation processing on the content stream to obtain at least one of the corresponding translated texts.

[0132] Furthermore, the first type of intelligent agent 501 and the second type of intelligent agent 502 are configured to perform different processing on the content stream.

[0133] For example, fact detection is performed on the content stream to obtain the detection result, which specifically includes: searching the content to be detected in the content stream on the Internet or a preset database, and determining the detection result based on the matching result between the search result and the content to be detected. The result obtained includes correct, incorrect or partially correct.

[0134] Taking the converted text obtained from the target data as an example, the content stream is explained as follows: A first-type agent 501 is configured to perform fact detection on the converted text to obtain the detection result, and a second-type agent 502 is configured to extract key words and phrases from the converted text and generate corresponding reference background information based on these key words and phrases. Therefore, during communication between a user and others, if the statements expressed by others are declarative sentences or judgments about a certain matter, the terminal device 200 can analyze the content stream to determine that the first-type agent 501 is the current target agent 504, and use the first-type agent 501 to perform fact detection on the converted text. If the statements expressed by others contain technical terms or high-frequency words and phrases, the terminal device 200 can analyze the content stream to determine that the second-type agent 502 is the current target agent 504, and use the second-type agent 502 to extract technical terms or high-frequency words and phrases from the content stream as key words and phrases, and generate reference background information based on these key words and phrases.

[0135] Furthermore, in some embodiments, the agent also includes a third agent 503, which is different from the first agent 501 and the second agent 502. The first agent 501, the second agent 502 and the third agent 503 are configured to perform different processing on the content stream, thereby enabling diverse agent selection based on the actual content of the content stream.

[0136] Therefore, this framework can flexibly adapt to the target agent 504 based on the content stream received by the near-eye display device 100, make full use of the target agent 504 to generate auxiliary information, and effectively improve the conversation assistance effect during the user conversation process.

[0137] Especially in scenarios such as negotiation and debate, the near-eye display device 100 can flexibly provide relevant auxiliary information based on the actual communication between the user and others, thus bringing great assistance.

[0138] In some embodiments, the intelligent agent includes at least a first type of intelligent agent 501 and a second type of intelligent agent 502 that is different from the first type of intelligent agent 501; in some embodiments, the first type of intelligent agent may correspond to a general intelligent agent and the second type of intelligent agent may correspond to an active intelligent agent.

[0139] Based on the analysis results of the content flow, target agent 504 is identified from the agents, including:

[0140] According to the order in which multiple content streams are input to the terminal device 200, the first type of intelligent agent 501 and the second type of intelligent agent 502 alternately present corresponding auxiliary information as target intelligent agents 504.

[0141] Specifically, the terminal device 200 is equipped with at least two intelligent agents. When the near-eye display device 100 transmits multiple content streams to the terminal device 200 or the transmitted content streams include multiple sub-contents, the terminal device 200 selects a first type of intelligent agent 501 and a second type of intelligent agent 502 as target intelligent agents 504 to alternately process the content streams and realize the alternating presentation of corresponding auxiliary information. This gives the intelligent agents enough time to process the content, so that the near-eye display device 100 can receive the auxiliary information in a timely manner and perform corresponding display actions during the user session. Moreover, assigning processing tasks to different intelligent agents can reduce the risk of a single intelligent agent crashing due to excessive load.

[0142] For example, when the content stream transmitted from the near-eye display device 100 to the terminal device 200 includes sub-contents W1, W2, W3, and W4 arranged sequentially according to time, the terminal device 200 uses a first type of intelligent agent 501 to generate auxiliary information corresponding to sub-content W1 and auxiliary information corresponding to sub-content W3, and uses a second type of intelligent agent 502 to generate auxiliary information corresponding to sub-content W2 and auxiliary information corresponding to sub-content W4.

[0143] In some implementations, the terminal device 200 is used for:

[0144] When the received content stream includes converted text, the converted text is analyzed using an analysis model to obtain the textual semantics of the converted text;

[0145] Based on the semantics of the text and the functions of the agent, target agents 504 were selected whose functions matched the semantics of the text.

[0146] Specifically, the terminal device 200 can first analyze the semantics of the converted text using an analysis model, and then match the target agent 504 in the agent based on the semantics. For example: when the converted text includes declarative sentences appearing in the conversation, an agent that can be semantically matched is used to perform fact detection on the converted text; or, when the converted text includes statements corresponding to the interaction object, an agent that can be semantically matched is used to generate recommended response information; or, when the converted text includes professional terms or high-frequency words and phrases appearing in the conversation, an agent that can be semantically matched is used to extract key words and phrases from the content stream and generate reference background information based on the key words and phrases; or, when the converted text includes foreign language statements appearing in the conversation, an agent that can be semantically matched is used to perform translation processing.

[0147] like Figure 7As shown, in some embodiments, the terminal device 200 is used for:

[0148] When the received content stream includes converted text, the converted text is input into each agent so that the agent can output the corresponding agent output result based on the converted text;

[0149] When the output of the agent meets the preset conditions, the corresponding agent is determined to be the target agent 504.

[0150] Specifically, the terminal device 200 can also input the converted text into each agent separately, select the target agent 504 based on the output of the agent, and use the output of the target agent 504 as auxiliary information.

[0151] For example, one could select the agent that outputs results within a preset time as the target agent 504, or one could verify the output results of the agents and select the agents whose output results pass the verification as the target agent 504. It should be understood that the above methods can eliminate the process of using an analysis model to analyze and transform text, thus speeding up the selection of the target agent 504 and the generation of auxiliary information.

[0152] In some embodiments, the controller is also configured to determine that a second intelligent agent needs to be included when the communication rate during the session reaches a preset value. For example, if multiple conversation objects (such as users A, B, C, and D) participate in the conversation in the conversation content stream, and the communication rate is 2 objects' content streams or statements appear every 1 second, then the user is reminded and the second intelligent agent is automatically added. By actively adding the second intelligent agent through the increase in the frequency of the conversation, the processing power of a single intelligent agent can be effectively distributed, which is beneficial to improving the efficiency of auxiliary information presentation.

[0153] In some embodiments, when an increase in the number of participants is detected in the conversation content stream, an additional agent is added. Other embodiments may also detect the emotional state of the conversation content stream. For example, the content stream can be input into a large language model to determine the emotional state of the participants in the conversation. When the content stream indicates that the participant's emotional state is angry, the speed at which the agent presents auxiliary information is increased accordingly; when the content stream indicates that the participant's emotional state is happy, the speed at which the agent presents auxiliary information is decreased accordingly; when the content stream indicates that the participant's speaking speed is slow, the speed at which the agent presents auxiliary information is decreased accordingly; and when the content stream indicates that the participant's speaking speed is fast, the speed at which the agent presents auxiliary information is increased accordingly.

[0154] In some embodiments, the time interval between different agents generating auxiliary information does not exceed 2 seconds, such as 50ms, 0.5s, 1s, etc. The presentation speed of auxiliary information on near-eye display devices is slower than that on terminal devices.

[0155] Figure 14 This is a schematic diagram illustrating the proactive interaction of auxiliary information from different intelligent agents in a chat scenario provided by an embodiment of this application. Both the first and second types of intelligent agents can generate proactive prompts and auxiliary information. The two agents can convert different content streams to obtain auxiliary information. It is evident that the auxiliary information provided by the second and first types of intelligent agents is different. Specifically, the second type of intelligent agent infers the intent (orientation) within the conversation content, providing corresponding red auxiliary information, while the first type of intelligent agent provides corresponding green auxiliary information. The second type of intelligent agent can infer the conversation intent or orientation based on the content stream. The auxiliary information generated by the first and second types of intelligent agents can be presented on the near-eye display device 100 and / or the terminal device 300. Users can also choose to have any one or a combination of the auxiliary information generated by the first and second types of intelligent agents presented on the near-eye display device 100. For users who can see this content (e.g., users wearing near-eye display devices), immediate proactive feedback can be provided based on the conversation content stream, offering more efficient guidance and effectively improving the communication efficiency of the conversation. It is understandable that the use of user avatar icons to represent different agents and conversation objects in the above-mentioned related figures is merely for the convenience of illustrating the content flow. In practice, the icons in the figures can be flexibly designed according to the agents and conversation objects.

[0156] In summary, this application provides a near-eye display device 100, including: a frame 10 for connecting an optical lens 80, the frame 10 having an ambient side 61 and a near-eye side 62; a wearing part 20 connected to the frame 10 for wearing on a user and projecting the optical lens 80 into the user's field of vision; a near-eye display module 30 connected to the frame 10 and / or the optical lens 80; a communication module 40 for communicating with a terminal device 200; and a controller 50 communicating with the near-eye display module 30 and the communication module 40, for: receiving a content stream output by the terminal device 200, the content stream being the target data of the acquired target object processed by the terminal device 200. The content is obtained through line conversion; auxiliary information is obtained by the terminal device 200 analyzing at least part of the content in the content stream, determining the target intelligent agent 504 from the intelligent agents based on the analysis results of the content stream, and using the target intelligent agent 504 to process the content stream; the near-eye display module 30 is controlled to display the auxiliary information matching the content stream to the near-eye side 62 based on the first attribute information of the content stream and the second attribute information of the auxiliary information; this application can flexibly adapt the target intelligent agent 504 based on the content stream received by the near-eye display device 100, make full use of the target intelligent agent 504 to generate auxiliary information, and effectively improve the conversation assistance effect during the user conversation process.

[0157] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in this application. These modifications or substitutions should all be covered within the scope of protection of this application, and the scope of protection of this application should be determined by the scope of the claims.

[0158] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, as a fixed connection, a detachable connection, or an integral connection; a mechanical connection or an electrical connection; a direct connection or an indirect connection via an intermediate medium; or a connection within two elements. Those skilled in the art will understand the specific meaning of the above terms in this application based on the specific circumstances. As used in this specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a," "an," and "the" are intended to include the plural forms. It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system.

Claims

1. A near-eye display device, characterized in that, The near-eye display device includes: A frame for connecting optical lenses, the frame having an opposing ambient side and a near-eye side; The wearing part is connected to the frame and is used to be worn by the user and to project the optical lens into the user's field of vision. A near-eye display module, connected to the frame and / or the optical lens; A communication module, used for communication connection with terminal devices; The controller, which is communicatively connected to the near-eye display module and the communication module, is used for: Receive the content stream output by the terminal device, wherein the content stream is the content obtained by the terminal device from the target data of the acquired target object; The auxiliary information is obtained by the terminal device analyzing at least a portion of the content stream, determining a target intelligent agent from the intelligent agents based on the analysis results of the content stream, and using the target intelligent agent to process the content stream. Based on the first attribute information of the content stream and the second attribute information of the auxiliary information, the near-eye display module is controlled to display the auxiliary information matching the content stream to the near-eye side.

2. The near-eye display device as described in claim 1, characterized in that, The near-eye display module includes a display interface opposite to the near-eye side, and the controller is further configured to: The association between the content stream and the auxiliary information is determined based on the first attribute information and the second attribute information; Based on the aforementioned association, the near-eye display module is controlled to display the content stream and the auxiliary information associated with the content stream on the display interface.

3. The near-eye display device as described in claim 2, characterized in that, The content stream includes at least the converted text obtained by converting the acquired speech data of the target object. The converted text and the auxiliary information associated with the converted text have the same text attributes on the display interface. The text attributes include, but are not limited to, at least one of text color, text font, and text size. And / or, The content stream and the auxiliary information associated with the content stream are displayed in the same area of ​​the display interface.

4. The near-eye display device as described in claim 1, characterized in that, The controller is also used for: The source information of the auxiliary information is determined based on the attribute information of the auxiliary information. The source information includes at least one of the source object corresponding to the auxiliary information and the target intelligent agent corresponding to it, wherein the content stream output by the source object corresponds to the auxiliary information. The near-eye display module is controlled to associate and display the auxiliary information with the same source information.

5. The near-eye display device as described in claim 4, characterized in that, The controller is also used for; Based on the source object, determine the first display information corresponding to the first source information and the second display information corresponding to the second source information from the auxiliary information, wherein the first source information and the second source information are different source information; The near-eye display module is controlled to display the first display information in a first area and the second display information in a second area; Wherein, the first region and the second region are different regions on the display interface; The first object icon corresponding to the first source information is displayed in the first area; the second object icon corresponding to the second source information is displayed in the second area.

6. The near-eye display device as described in claim 4, characterized in that, The content stream includes at least the converted text obtained by converting the target data of the acquired target object. The converted text to be displayed with the same source information has the same text attributes on the display interface. The text attributes include, but are not limited to, at least one of text color, text font, and text size.

7. The near-eye display device as described in claim 1, characterized in that, The controller is also used for: When it is determined that there is no matching auxiliary information in the content stream, the near-eye display module is controlled not to display the auxiliary information corresponding to the converted text.

8. The near-eye display device as described in claim 1, characterized in that, The controller is also used for: The content stream is output to the terminal device so that the terminal device generates matching auxiliary information based on the content stream; The controller is further configured to: receive the auxiliary information output by the terminal device; Receive the content stream output by the first terminal device; The content stream is output to a second terminal device, so that the second terminal device generates matching auxiliary information based on the content stream; Receive the auxiliary information output by the second terminal device; The first terminal device and the second terminal device are different terminal devices.

9. The near-eye display device as described in claim 1, characterized in that, The near-eye display device further includes a data acquisition module that is communicatively connected to the controller. The data acquisition module is configured to acquire data, and the data acquired by the data acquisition module includes at least one of voice, text, and video. The controller is also used for: The data acquisition module is controlled to acquire target data of the target object and receive the target data. The target data is transmitted to the terminal device via the communication module, so that the terminal device can convert the target data to obtain the content stream; Receive the content stream output by the terminal device.

10. The near-eye display device as claimed in claim 1, characterized in that, The intelligent agent includes at least a first type of intelligent agent and a second type of intelligent agent that is different from the first type of intelligent agent; The first type of intelligent agent is configured to perform: fact detection on the content stream to obtain a detection result; generate recommended response information corresponding to the wearer of the near-eye display device based on the content stream; extract key words and phrases from the content stream and generate corresponding reference background information based on the key words and phrases; and perform translation processing on the content stream to obtain at least one of the corresponding translated texts. The second type of intelligent agent is configured to perform: fact detection on the content stream to obtain a detection result; generate recommended response information corresponding to the wearer of the near-eye display device based on the content stream; extract key words and phrases from the content stream and generate corresponding reference background information based on the key words and phrases; perform translation processing on the content stream to obtain at least one of the corresponding translated texts; and perform intent recognition on the conversation content stream. Furthermore, the first type of intelligent agent and the second type of intelligent agent are configured to perform different processes on the content stream.

11. The near-eye display device as claimed in claim 1, characterized in that, The intelligent agent includes at least a first type of intelligent agent and a second type of intelligent agent that is different from the first type of intelligent agent; The step of determining the target agent from the agents based on the analysis results of the content stream includes: According to the order in which the multiple content streams are input to the terminal device, the first type of intelligent agent and the second type of intelligent agent respectively serve as the target intelligent agent to alternately present corresponding auxiliary information.

12. The near-eye display device as described in claim 1, characterized in that, The terminal device is used for: When the received content stream includes converted text, the converted text is analyzed using an analysis model to obtain the text semantics of the converted text; Based on the semantics of the text and the functions of the agent, the target agent that matches the semantics of the text is selected from among the agents.

13. The near-eye display device as described in claim 1, characterized in that, The terminal device is used for: When the received content stream includes converted text, the converted text is input into each agent so that the agent outputs the corresponding agent output result based on the converted text; When the output result of the intelligent agent meets the preset conditions, the corresponding intelligent agent is determined to be the target intelligent agent.

Images