Permanent connection system for generative ai
The wearable terminal device streamlines communication with generative AI by transmitting real-time video and audio data, addressing the inefficiency of manual summarization and enabling immediate, personalized information provision.
Patent Information
- Application Number
- JP2023086492
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-25
- Publication Date
- 2024-12-05
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing generative AI systems require users to actively summarize and communicate complex experiences, which is time-consuming and inefficient, hindering immediate and accurate information exchange.
A wearable terminal device continuously monitors video and audio data, transmitting it to a server for real-time processing by generative AI, eliminating the need for manual text summarization and enabling immediate information provision tailored to the user's situation.
This system allows for instantaneous receipt of personalized knowledge and know-how, enhancing user experience by providing timely and relevant information and advice through continuous learning from daily life interactions.
Smart Images

Figure 2024169211000001_ABST
Abstract
Description
[Technical field]
[0001] An embodiment of the present invention relates to a generative AI system, a server, and a terminal device that are constantly connected to a wearable terminal. [Background technology]
[0002] Generative AI is a large-scale language model that has learned large amounts of information, such as web information, and by sending prompts to the generative AI as requirements and receiving responses from the generative AI, we can easily obtain vast amounts of knowledge. It is also possible to have the generative AI learn additional information, and obtain responses that reflect that additional information. Generative AI with such capabilities is a very useful tool for inquiring about knowledge and know-how. [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] "DEFiDE, which provides the job-based HR cloud service JOB Scope, provides functions for converting 1-on-1 interviews into text and summarizing them and analyzing the interviewee's emotions. Uses AI to analyze and digitize important points during feedback interviews and the emotions of the interviewee" (https: / / prtimes.jp / main / html / rd / p / 000000024.000004868.html) Confirmed on May 2023. Summary of the Invention [Problem to be solved by the invention]
[0004] In order to receive knowledge and know-how from the generative AI, it is necessary to actively communicate the requirements to the generative AI in some way. Depending on the content, these requirements can be diverse, ranging from simple inquiries about the meaning of words to advanced advice based on the experiences of individuals in their daily lives. In order to communicate advanced requirements, it is necessary to summarize the experience and convey it to the generative AI, which takes time and effort. In our daily lives, we encounter various situations, and if we do not have the knowledge and know-how to appropriately correspond to the flow of conversations, etc., there is a high need to utilize the generative AI. However, since it is necessary to summarize the experience, it is currently not possible to immediately convey the requirements to the generative AI. [Means for solving the problem]
[0005] In order to eliminate the need for the user to summarize and explain their previous experiences when asking a question to the generating AI and to achieve accurate information sharing, the wearable terminal device constantly captures video and audio and transmits them to the generating AI. This eliminates the need for the user to transmit summarized text information to explain the situation to the generating AI, and allows the provision of knowledge, know-how, and other information appropriate to the situation at hand to be provided instantly through interactions between the wearable terminal device and the AI on the server. Effect of the Invention
[0006] This invention makes it possible to transmit images and sounds from a user's daily life to a generation AI in real time using a wearable terminal device. This saves the user the trouble of transmitting textual information to explain the situation to the generation AI, and enables the user to instantly receive information such as knowledge and know-how according to the situation. This system is expected to improve the quality of life of users, as it can efficiently provide information that is useful in various situations such as conversations and problem solving. In addition, generative AI can provide more personalized advice and support by continuously learning from information obtained in the user's daily life, allowing users to obtain information and services that are tailored to them and receive effective assistance in dealing with various challenges in daily life. As described above, by combining a wearable terminal device with generative AI, the present invention provides a system that can efficiently and appropriately provide information and support to users in their daily lives, thereby contributing to improving the quality of users' lives. [Brief description of the drawings]
[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of an overall configuration according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of a server and a wearable information terminal device according to the embodiment. [Diagram 3] FIG. 3 is a sequence diagram showing an example of the operation of the server and the wearable information terminal device according to the embodiment. [Figure 4] FIG. 4 is a sequence diagram showing an operation example in which a person identification function and an instruction discrimination function are added to the information conversion unit 1 of the server according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0008] Hereinafter, an embodiment will be described with reference to the drawings. Fig. 1 is a diagram showing an outline of the overall configuration of a system according to an embodiment. Note that numbers and symbols in the diagram are shown in parentheses below. To implement the present invention, a user uses a wearable terminal device (e.g., smart glasses, smart watch, smartphone, wireless earphones) in daily life. This wearable information terminal device is equipped with a camera, microphone, speaker, and LCD panel for acquiring and outputting video and audio. The video and audio data acquired by the wearable terminal device is transmitted to a server via the Internet using wireless communication technology (e.g., Bluetooth, Wi-Fi).
[0009] An image captured by input unit 1 (101) of information terminal device 1 (10) is received by receiving unit 3 (501) of information terminal device 5 (50) from transmitting unit 1 (102) by communication technology such as Bluetooth, and is transmitted from transmitting / receiving unit (502) of information terminal device 5 (50) to information conversion unit 1 (601) of server (60) via the Internet using Wi-Fi or a mobile phone line.
[0010] The voice captured by the input unit 2 (201) of the information terminal device 2 (20) is received by the receiving unit 3 (501) of the information terminal device 5 (50) via a communication technology such as Bluetooth from the transmitting unit 2 (202), and is transmitted from the transmitting / receiving unit (502) of the information terminal device 5 (50) to the information conversion unit 1 (601) of the server (60) via the Internet using Wi-Fi or a mobile phone line.
[0011] The received video data is converted into text information using image captions, and the audio data is converted into text information using automatic voice recognition technology. The converted text information is transmitted via the receiver / transmitter unit (502) of the information terminal device 5 (50) or directly to the server's analysis unit (602), where it is learned by the generation AI.
[0012] The generation AI provides appropriate information and advice based on the user's daily life experience information learned by the analysis unit (602) and instructions from the user, which will be described later. The analysis unit (602) of the server transmits messages addressed to the user from the learned generation AI via the receiver / transmitter unit (502) of the information terminal device 5 (50) or directly to the information conversion unit 2 (603) of the server.
[0013] The information conversion unit 2 (603) converts the character information created by the analysis unit (602) into voice information and transmits it to the transmitting / receiving unit (502) of the information terminal device 5 (50).
[0014] The receiver / transmitter unit (502) transmits the text information from the analyzer unit (602) and the voice information from the information converter 2 (603) to the transmitter 3 (503).
[0015] The transmission unit 3 (503) transmits the character information from the analysis unit (602) and the voice information from the information conversion unit 2 (603) by a communication technology such as Bluetooth.
[0016] The voice information transmitted from the transmission unit (503) is received by the reception unit 1 (302) of the information terminal device 3 (30) and output as voice from the speaker of the earphone by the output unit 1 (301).
[0017] The text information transmitted from the transmission unit (503) is received by the reception unit 2 (402) of the information terminal device 4 (40) and output as text on the screen of a smartphone or the like by the output unit 2 (401). This allows the user to receive knowledge and know-how according to the situation in real time.
[0018] 2 is a block diagram showing an example of the configuration of a server and a wearable information terminal device according to an embodiment. The information terminal device 1 is a device that can be worn at all times and is equipped with a small camera, and may be a smart glass, a pendant, a necklace, or a tie pin.
[0019] The information terminal device 2 is a device that can be worn at all times and is designed to accommodate a small microphone, and may be smart glasses, a smartphone, a pendant, a necklace, a tie pin, or wireless earphones.
[0020] The information processing terminal 3 is a device capable of outputting audio, and may be a wireless earphone or smart glasses.
[0021] The information processing terminal 4 is a device capable of displaying characters, and may be smart glasses, a smart watch, or a smartphone.
[0022] The information processing terminal 5 is a wearable information terminal device to which application programs can be downloaded, and may be a smart watch or a smartphone.
[0023] 2 shows an embodiment in which information terminal device 1 and information terminal device 2, and information terminal device 4 and information terminal device 5 are integrated. Each of these information processing devices 1 to 5 may be implemented as an independent device. Also, any pattern of integration may be implemented, in which case transmission / reception functions present in the integrated parts of transmitters 1 to 3 and receivers 1 to 3 between the information processing devices are not required. Cases in which any two of the information processing devices are integrated, three are integrated, four are integrated, or all are integrated may also be considered.
[0024] 3 is a sequence diagram showing an example of the operation of a server and a wearable information terminal device according to the embodiment. First, the information terminal device 1 may be smart glasses. An image is input from a camera installed in the smart glasses to the input unit 1 (ST10), and the information is transmitted from the transmission unit 1 to the information terminal device 5 by wireless communication such as Bluetooth or by wire (ST11).
[0025] The information terminal device 2 may be smart glasses. Voice is input to the input unit 2 through a microphone installed in the smart glasses (ST20), and the transmission unit 2 transmits the information to the information terminal device 5 wirelessly by Bluetooth or via a wired connection (ST21).
[0026] In information processing terminal 5, receiving section 1 receives the video information transmitted from information terminal device 1 and the audio information transmitted from information terminal device 2 (ST50), and transmits the information to the receiving / transmitting section of information processing terminal 5. The receiving / transmitting section calls information conversion section 1 (ST51), and transmits the video information and audio information to information conversion section 1 of the server via the Internet.
[0027] The information conversion section 1 of the server receives image information and audio information (ST60), and converts the image information into image captions and the audio information into text information using automatic voice recognition (ST61). The text information acquired in this way can be linked to the transmitting / receiving section of the information terminal device 5 via the Internet, or directly to an analysis section in the server. In the former, the transmitting / receiving section of the information terminal device 5 acquires the text information (ST52), calls the analysis section of the server (ST53), and transmits the information to the analysis section of the server via the Internet.
[0028] In the analysis section of the server, the generation AI learns the character information received via the receiver / transmitter section of the information terminal device 5 or directly linked from the information conversion section 1 (ST63-ST64). After learning, the generation AI actively creates appropriate messages based on the user's experience or messages in response to instructions from the user (described later) (ST65). These messages may include recommendations according to the user's preferences, or warnings or alerts according to the situation of the scene. The message information acquired in this way may be linked to the receiver / transmitter section of the information terminal device 5 via the Internet, or directly linked to the information conversion section 2 in the server. In the former case, the receiver / transmitter section of the information terminal device 5 acquires the message information (ST54), calls the information conversion section 2 of the server (ST55), and transmits the information to the information conversion section 2 of the server via the Internet.
[0029] The information conversion unit 2 of the server receives the message information linked via the transmitting / receiving unit of the information terminal device 5 or directly from the analysis unit (ST66) and converts it into voice information by AI (ST67). The voice information and the message information thus acquired are linked to the transmitting / receiving unit of the information terminal device 5 via the Internet (ST56).
[0030] The receiver / transmitter section of the information terminal device 5 transmits the message created by the generation AI and the voice information converted from it to the transmitter section 2. The transmitter section 2 transmits the information to the information terminal device 3 and the information terminal device 4 by wireless communication such as Bluetooth or by wire (ST57).
[0031] The receiving section 1 of the information processing terminal 3 receives the audio information transmitted by the transmitting section 2 of the information processing terminal 5 (ST31), and outputs the audio as sound from the speaker of the wireless earphone (ST30).
[0032] The receiving unit 2 of the information processing terminal 4 receives the message information transmitted by the transmitting unit 2 of the information processing terminal 5 (ST41) and outputs it to the screen of the smartphone (ST40). The message may be displayed as text information on the glass surface of the smart glasses.
[0033] The flow of receiving user instructions with improved accuracy by adding a function to the information conversion unit 1 of the server in FIG. 4 will be described. Image information and audio information are received from the receiver / transmitter of the information terminal device 5 (ST70), and the image information is converted to image caption and the audio information is converted to text information by automatic voice conversion (ST71). After this step, steps ST72 to ST75 are added. Based on previously learned information, a person is identified from the characteristics of the image information and audio information (ST72). In addition, it is determined whether the person making the voice utterance is a user who owns this system (ST73). If it is a user who owns the system, it is determined whether the spoken words are a password for instructions. If it is a password, an instruction flag is set (ST75). This person information and instruction flag can be transmitted to the analysis unit of the server via the receiver / transmitter of the information terminal device 5, or directly without the receiver / transmitter being involved.
[0034] This additional function allows the user to give instructions to the generation AI from a wearable information terminal device. In other words, information is transmitted to the generation AI as one of the voice information in the above embodiment. The user speaks instructions to the generation AI through the microphone of the information terminal device 2, and the generation AI recognizes that they are the user's words from the characteristics of the voice waveform, and determines that they are instructions to the generation AI from the content of the utterance. For example, an embodiment may be one in which the words "teach AI" are determined as a password for the user's instructions, and the content of the utterance following the words is recognized as an instruction to the generation AI.
[0035] When instructions are received from the user, the generation AI transmits its answer from the server's analysis unit through the receiver-transmitter unit of information terminal device 5, the server's information conversion unit 2, the receiver-transmitter unit and transmitter unit 2 of information terminal device 5, to the receiver unit 1 and output unit 1 of information terminal device 3, where it is transmitted to the user as voice information, and then to the receiver unit 2 and output unit 2 of information terminal device 4, where it is transmitted to the user as message information.
[0036] As described above, in one embodiment of the present invention, a system is constructed that transmits video and audio data to a generating AI through a wearable terminal device and allows users to obtain appropriate information and advice from the generating AI. This enables users to utilize the generating AI efficiently and in a timely manner in various situations in their daily lives.
[0037] Furthermore, in this embodiment, the user can send questions and instructions to the generation AI through the wearable terminal device. This allows the user to obtain information proactively in addition to information provided by the generation AI. In addition, the generation AI can automatically provide the user with warnings and recommendations based on the user's daily life experience information that it has learned, in addition to information based on instructions from the user.
[0038] In the embodiment of the present invention, the wearable terminal device and the generation AI can be linked with other smart devices. That is, the smart device can be used as the information processing terminal 4, and various functions can be performed on the smart device according to information from the generation AI. For example, in cooperation with a smart home system or a smartphone application, it is possible to automatically optimize settings in the home, such as lighting and temperature control, based on information and advice from the generation AI. Alternatively, when the generation AI detects that the user has fallen asleep from an image or voice of the user, it is also possible to change the lighting or the inclination of the bed to a state suitable for sleep.
[0039] In an embodiment of the present invention, a user can customize the settings of a wearable terminal device or application. For example, the user can stop the acquisition of video and audio data at a specific time or location, or adjust the frequency of information provided by the generation AI and the type of information provided. This allows the user to optimize the system to suit their needs and circumstances and achieve a comfortable usage experience.
[0040] As described above, the present invention combines a wearable terminal device and a generating AI to transmit the contents of an individual's daily life experiences to the generating AI in real time, and the generating AI learns more about the environment surrounding the user. In addition, by learning the user's behavior, knowledge about the user is also accumulated, making it possible to provide recommendations and messages that are more personalized according to the user's hobbies and preferences. This is expected to make it easier and more efficient for users to obtain and use information, improving the quality of their daily lives.
[0041] In this embodiment, the generation AI continuously learns the user's daily life experience information and is able to provide more appropriate information and advice based on the results. The performance of the generation AI improves as the learning progresses, allowing it to respond more adaptively to the user's needs.
[0042] Regarding this invention, the claims include not only method inventions, but also processing programs, servers, and system inventions.
[0043] In addition, by describing an information terminal device 5 having a receiving unit 3, a receiving / transmitting unit, and a transmitting unit 3 as a client device and claiming it separately, an embodiment in which only the receiving unit 3, the receiving / transmitting unit, and the transmitting unit 3 of the information terminal device 5 are provided and everything else is provided by utilizing materials provided by others, which ultimately achieves the same effect as the present invention, is also included in the scope of the claims of the present invention.
[0044] In the embodiment of the present invention, the privacy and security of the user are important factors. Video and audio data transmitted from the wearable terminal device is encrypted and transmitted to the server via a secure connection. Personal information that requires confidentiality is masked and transmitted to the generation AI, and is excluded from the learning subject. The generation AI on the server learns the user's experience information without accessing personal information. Furthermore, privacy is protected by notifying the user of information and advice provided by the generation AI in a form that does not include personal information.
[0045] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims.
Claims
1. A method for providing a user with the results of an analysis of detected video and audio by an AI generator, the method comprising the steps of: (a) providing a system for providing a user with the results of an analysis of detected video and audio by an AI generator; (b) providing a system for providing a user with useful information by providing a system that provides ...
2. The present invention relates to a wearable information terminal device that includes an input unit that grasps video images installed in a wearable information terminal device that is carried by a user, a transmission unit that transmits the video information from the information terminal device via a wired or wireless line, an input unit that grasps audio images installed in a wearable information terminal device that is carried by a user, a transmission unit that transmits the audio information from the information terminal device via a wired or wireless line, a reception unit that receives the video information and audio information transmitted from the information terminal device and the information terminal device installed in a wearable information terminal device, and a reception / transmission unit that transmits and receives information from the information terminal device to a server via an internet line, an information conversion unit that converts the video information received from the reception / transmission unit into text information by an image caption in the server and converts the audio information into text information by automatic voice conversion, and a transmission unit that receives the text information converted by the information conversion unit and the video information and audio information before conversion from the information conversion unit via the reception / transmission unit or directly from the information conversion unit, and causes a generation AI to learn the information, and then transmits the learned generation AI to the user. a transmission unit 3 for transmitting the character information generated by the analysis unit and the voice information converted by the information conversion unit 2 to a wearable information terminal device 3 carried by the user; a reception unit 1 for receiving the voice information transmitted by the transmission unit 3 at a wearable information terminal device 3 carried by the user; an output unit 1 for outputting the voice information received by the reception unit 1; a reception unit 2 for receiving the character information transmitted by the transmission unit 3 at a wearable information terminal device 4 carried by the user; and an output unit 2 for outputting the character information received by the reception unit 2, wherein the system responds to the user with the results of an analysis of detected video and audio by a generating AI, the system having a function of constantly connecting input and output of video and audio information by using a wearable information terminal device, and the constant connection allows input and output of information at all times, not just each time it is used.
3. 3. The method according to claim 1, wherein the information conversion unit 1 of the server has a function of identifying a person from an image and a voice, and further has a function of transmitting the identified person information to the analysis unit of the server through the transmitting / receiving unit of the information terminal device 5 or directly without the intermediate transmitting / receiving unit.
4. The method according to any one of claims 1 to 3, wherein the user can give instructions to the generation AI from a wearable information terminal device.
5. The method according to any one of claims 1 to 3, characterized in that even if there is no instruction as claimed in claim 4, information is transmitted from the generating AI to the user.
6. A server device used in the system according to any one of claims 1 to 5.
7. A system that responds to a user with the results of an analysis of detected video and audio by a generating AI, comprising a receiving unit 3 that receives information from the information terminal device 1 and the information terminal device 2 according to claim 2, a receiving / transmitting unit that transmits and receives information to and from the information transmitting unit 1, analysis unit, and information transmitting unit 2 of the server, and a transmitting unit 3 that transmits information to the information terminal device 3 and the information terminal device 4, and that uses a wearable information terminal device to provide a function for constantly connecting the input and output of video and audio information, and that allows input and output of information at all times, not just each time it is used, with the client device characterized in that
8. A program for causing a computer to execute the processing method and apparatus according to any one of claims 1 to 7, and a computer-readable recording medium on which the program is recorded.
9. A system including a wearable information terminal device 1, a wearable information terminal device 2, a wearable information terminal device 3, a wearable information terminal device 4, and a wearable information terminal device 5 for implementing the method according to any one of claims 1 to 5.
10. 10. The system according to claim 9, wherein the wearable information terminal devices 1 and 2 are wearable glasses, the wearable information terminal device 3 is a wireless earphone, and the wearable information terminal devices 4 and 5 are smartphones.
11. 10. The system according to claim 9, wherein the five types of wearable information terminal devices 1 to 5 are different types of devices, or a part or all of the five types of devices are integrated into a wearable device.
12. The method, system, device, program or recording medium according to any one of claims 1 to 11, wherein the information terminal device 4 is a non-wearable smart system.
13. The method, system, device, program, or recording medium according to any one of claims 1 to 12, characterized in that the settings of the wearable terminal device and applications can be customized.
14. The method, system, device, program, or recording medium according to any one of claims 1 to 13, wherein the generating AI has the ability to adapt to the user's living situation or environment, or provides information tailored to the user's needs and preferences through interaction with the user.
15. The method, system, device, program, or recording medium according to any one of claims 1 to 14, wherein the generating AI improves the accuracy of the information provided based on feedback from the user.
