System

Abstract

To provide a system.SOLUTION: A system includes: means for analyzing a material stored in advance, creating a summary of the material, and translating it into a user's mother tongue; means for providing the user with the translated summary; means for acquiring utterances in a conference; means for making the acquired utterances a text by using a speech recognition technology; means for analyzing the text-converted utterances, generating a summary, and translating it into the user's mother tongue; and means for providing the user with the translated summary of the utterances.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] Previously, communication between different languages, obtaining information, and procedures such as visa applications were difficult due to language barriers, lack of information, complicated procedures, etc. In particular, foreigners staying in Japan have experienced great difficulties in understanding Japanese emails, grasping information at meetings, understanding Japanese customs, and procedures such as visa applications. [Means for solving the problem]

[0005] The present invention solves the above problems by providing a means to summarize emails, convert keywords in one's native language into Japanese sentences, summarize advance materials in one's native language, instantly summarize speeches of participants, instantly research difficult words, tell what day it is and what customs exist, generate appropriate guidance for visa applications and renewals in accordance with the latest visa and immigration rules, and support the preparation of documents required for applications. This makes it easier to communicate between different languages, obtain information, and carry out procedures such as visa applications. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12]FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. [Figure 19] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system according to the embodiment 2 of the second embodiment in the case where an emotion engine is combined. [Figure 20] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when combined with an emotion engine. [Figure 21] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the embodiment 3 of the form example 3 in the case where an emotion engine is combined. [Figure 22] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0023] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] An embodiment of the present invention provides a system as follows. First, as a means for summarizing emails, the body of the email is analyzed, important information is extracted, and a summary is generated. For example, important keywords and phrases are extracted from the body of the email, and a summary is generated based on these. Next, as a means for converting native language keywords into Japanese sentences, native language keywords entered by the user are analyzed, and corresponding Japanese sentences are generated. For example, when keywords such as "meeting" or "materials" are entered, corresponding Japanese sentences are generated.

[0029] "Example 2"

[0030] Furthermore, as a means of summarizing advance materials into the user's native language, materials distributed in advance are analyzed and summaries are generated in the user's native language. For example, materials such as meeting agendas and minutes are analyzed and summaries are provided in the user's native language. Also, as a means of instantly summarizing participants' speech, speech during a meeting is analyzed in real time and a summary is generated. For example, speech during a meeting is converted into text using speech recognition technology, and that text is analyzed to generate a summary.

[0031] "Example 3"

[0032] Furthermore, as a means of instantly researching difficult words, it analyzes words that the user does not understand and looks up their meaning and usage. For example, it looks up technical terms and abbreviations from sources on the Internet and provides the results to the user. Also, as a means of teaching users what day it is and what customs are associated with it, it looks up Japanese holidays, anniversaries, seasonal events, etc. and provides that information to the user. For example, it looks up what day it is, the customs and events that take place on that day, etc. and provides that information to the user.

[0033] "Example 4"

[0034] Finally, the system generates appropriate guidance for visa applications and renewals in line with the latest visa and immigration rules, and as a means of supporting the preparation of documents required for application, generates a list of required documents based on information input by the user and provides instructions on how to prepare each document. For example, the system generates a list of required documents based on information such as the user's visa type, period of stay, and purpose of application, and provides instructions on how to prepare each document and where to submit it.

[0035] The process flow of each embodiment will be described below.

[0036] "Example 1"

[0037] Step 1: A user email is sent to the system.

[0038] Step 2: The system analyzes the body of the email and extracts key information.

[0039] Step 3: Based on the extracted information, the system generates a summary of the email.

[0040] Step 4: The generated summary is sent to the user, who reviews the summary.

[0041] "Example 2"

[0042] Step 1: The user enters keywords in their native language into the system.

[0043] Step 2: The system analyzes the entered keywords.

[0044] Step 3: The system generates Japanese sentences that correspond to the analyzed keywords.

[0045] Step 4: The generated Japanese text is sent to the user, who confirms the text. "Example 3"

[0046] Step 1: Pre-distributed materials are sent to the system.

[0047] Step 2: The system analyzes the material and extracts key information.

[0048] Step 3: Based on the extracted information, the system generates a summary of the material in the native language.

[0049] Step 4: The generated summary is sent to the user, who reviews the summary.

[0050] "Example 4"

[0051] Step 1: User's visa application information is entered into the system.

[0052] Step 2: The system will generate a list of required documents based on the information entered.

[0053] Step 3: Provide the user with the generated list and instructions on how to create each document.

[0054] Step 4: The user follows the instructions to prepare the documents and submit the visa application.

[0055] Example 1

[0056] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0057] In today's business environment, there is a need to quickly understand the contents of emails and efficiently extract important information. Also, to facilitate communication between different languages, technology is needed to convert keywords in one's native language into other languages. Furthermore, procedures for visa applications and renewals are complicated, and appropriate guidance and support for document preparation are required. A comprehensive system to solve these problems is needed.

[0058] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0059] In this invention, the server includes a means for summarizing the email, a means for converting native language keywords into Japanese sentences, a means for analyzing the email body using a generative AI model, extracting important information, and generating a summary, and a means for converting native language keywords into Japanese sentences using the generative AI model. This allows the user to quickly understand the content of the email and efficiently extract important information. It also allows smooth communication between different languages, and appropriately supports visa application and renewal procedures.

[0060] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[0061] The "means for converting keywords in one's native language into sentences in Japanese" refers to a means for analyzing keywords in one's native language input by a user and generating corresponding sentences in Japanese.

[0062] The "means for summarizing advance materials into the native language" refers to a means for analyzing materials provided in advance and generating a summary in the native language.

[0063] The "means for instantly summarizing participant speech" refers to a means for analyzing the speech of participants during a meeting or discussion in real time and generating a summary.

[0064] An "instant search tool for difficult words" is a means for instantly searching difficult words that appear in conversations or texts, and providing their meanings and related information.

[0065] A "means of teaching what day and customs it is today" is a means of providing information about events and customs associated with a particular date.

[0066] The "Visa Application and Renewal Support Tool" is a tool that generates guidance necessary for visa applications and renewals based on the latest visa and immigration rules, and supports the preparation of documents necessary for applications.

[0067] A "generative AI model" is a model that uses artificial intelligence technology to generate and analyze text.

[0068] A "prompt sentence" is an instruction sentence input to a generative AI model, which is an input sentence to obtain a specific output.

[0069] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0070] The present invention is a system including a means for summarizing emails and a means for converting keywords in a native language into Japanese sentences. A specific embodiment of this system will be described below.

[0071] Email summary method

[0072] The server retrieves emails received by the user. It uses a mail server (e.g., Gmail API or IMAP protocol) to retrieve new emails from the user's mailbox. It uses a natural language processing library such as Python's NLTK or spaCy to analyze the body of the retrieved email. This allows it to extract important keywords and phrases from the email body. It uses a generative AI model (e.g., OpenAI's (registered trademark) GPT-3 (registered trademark)) to generate a summary based on the extracted keywords and phrases. The generated summary is displayed in the user's email client.

[0073] Examples:

[0074] If the text of an email received by a user is "Tomorrow's meeting will discuss the progress of the new project. Please prepare materials.", the server retrieves this email and uses NLTK to extract keywords such as "meeting," "new project," "progress," and "materials." It then uses GPT-3 to generate a summary sentence that reads "Tomorrow's meeting will discuss the progress of the new project. Please prepare materials." and displays it in the user's email client.

[0075] Example prompt:

[0076] "Please summarize the body of the email below: In tomorrow's meeting, we will discuss the progress of the new project. Please prepare the materials."

[0077] A means of converting keywords in your native language into Japanese sentences

[0078] The device analyzes the native language keywords entered by the user and generates corresponding Japanese sentences. When the user enters native language keywords into the device, the device analyzes these keywords and creates a prompt sentence to input to the generative AI model. The prompt sentence is input into the generative AI model (e.g., OpenAI's GPT-3) to generate Japanese sentences. The generated Japanese sentences are displayed on the user's screen.

[0079] Examples:

[0080] If a user enters the keywords "meeting" and "documents" into the device, the device analyzes these keywords and creates a prompt sentence: "Generate a Japanese sentence containing the following keywords: meeting, documents." It then uses GPT-3 to generate the Japanese sentence "Please prepare documents for the meeting" and displays it on the user's screen.

[0081] Example prompt:

[0082] "Generate Japanese sentences containing the following keywords: meeting, materials"

[0083] In this way, the server and the terminal cooperate to generate summaries of the emails received by the user and convert keywords in the user's native language into Japanese sentences. This system allows the user to quickly understand the contents of the email and facilitates communication between different languages.

[0084] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0085] Email summary method

[0086] Step 1: Receiving email

[0087] The server obtains the email received by the user.

[0088] Input: New mail from the user's mailbox

[0089] Output: Body of retrieved email

[0090] What happens: The server uses a mail server (e.g. Gmail API or IMAP protocol) to retrieve new emails from the user's mailbox.

[0091] Step 2: Analyzing the email body

[0092] The server analyzes the body of the retrieved email.

[0093] Input: Body of the retrieved email

[0094] Output: Important keywords and phrases

[0095] How it works: The server uses natural language processing libraries such as Python's NLTK or spaCy to extract important keywords and phrases from the body of the email.

[0096] Step 3: Generate a summary

[0097] The server generates a summary based on the extracted keywords and phrases.

[0098] Input: Important keywords or phrases

[0099] Output: Generated summary

[0100] How it works: The server uses a generative AI model (e.g., OpenAI's GPT-3) to generate a summary based on the extracted keywords and phrases.

[0101] Step 4: Provide a summary

[0102] The server provides the generated summary to the user.

[0103] Input: Generated summary

[0104] Output: Summary text that will be displayed in the user's mail client

[0105] Specific operation: The server displays the generated summary in the user's email client.

[0106] A means of converting keywords in your native language into Japanese sentences

[0107] Step 1: Enter your keywords

[0108] The user inputs keywords in his / her native language into the terminal.

[0109] Input: Keywords in your native language (e.g. "meeting", "documents")

[0110] Output: The entered keyword

[0111] Specific operation: The user inputs a keyword in his / her native language into the input field of the terminal.

[0112] Step 2: Analyze keywords

[0113] The terminal analyzes the input keyword.

[0114] Input: The entered keyword

[0115] Output: A prompt to be fed to the generative AI model

[0116] Specific operation: The terminal analyzes the entered keywords and creates a prompt sentence saying, "Please generate Japanese sentences containing the following keywords: meeting, materials."

[0117] Step 3: Japanese sentence generation

[0118] The device inputs a prompt sentence into the generative AI model and generates a Japanese sentence.

[0119] Input: Prompt statement

[0120] Output: Generated Japanese sentences

[0121] Specific operation: The device inputs a prompt sentence into a generative AI model (e.g., OpenAI's GPT-3), which generates the Japanese sentence, "Please prepare materials for the meeting."

[0122] Step 4: Provide Japanese text

[0123] The terminal provides the generated Japanese text to the user.

[0124] Input: Generated Japanese sentences

[0125] Output: Japanese text displayed to the user on the screen

[0126] Specific operation: The terminal displays the generated Japanese text on the user's screen.

[0127] (Application example 1)

[0128] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0129] When drivers and passengers check their emails in autonomous vehicles, it is difficult to obtain information efficiently while ensuring driving safety. In addition, there are cases where appropriate translations are not made when converting keywords in one's native language into Japanese sentences. Furthermore, there is a lack of guidance on visa applications and renewals, making it difficult to prepare the documents required for application.

[0130] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0131] In this invention, the server includes a means for summarizing emails, a means for converting keywords in the native language into Japanese sentences, and a means for providing summaries in real time for efficiently checking emails in an autonomous vehicle. This makes it possible to efficiently check emails in an autonomous vehicle, and to quickly obtain important information while ensuring driving safety. In addition, by appropriately converting keywords in the native language into Japanese, user convenience is improved. Furthermore, by providing guidance on visa applications and renewals, it becomes easier to prepare documents required for applications.

[0132] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[0133] The "means for converting keywords in one's native language into sentences in Japanese" refers to a means for analyzing keywords in one's native language input by a user and generating corresponding sentences in Japanese.

[0134] The "means for summarizing advance materials into the native language" refers to a means for analyzing materials provided in advance and generating a summary in the native language.

[0135] The "means for instantly summarizing participant speech" refers to a means for summarizing in real time what is said by participants in a meeting or event.

[0136] An "instant lookup tool for difficult words" is a tool that instantly looks up difficult words and technical terms and provides their meanings and explanations.

[0137] A "means of teaching what day and customs it is today" is a means of providing information about events and customs associated with a particular date.

[0138] "Means of generating appropriate guidance on visa applications and renewals and supporting the preparation of documents required for applications" means means of providing guidance on visa applications and renewals and supporting the preparation of necessary documents based on the latest visa and immigration rules.

[0139] A "means for providing a real-time summary for efficiently checking emails within an autonomous vehicle" is a means for providing a real-time summary of emails when a driver or passenger checks emails within an autonomous vehicle.

[0140] The present invention relates to a system for enabling drivers and passengers in autonomous vehicles to efficiently check emails and convert keywords in their native language into Japanese sentences. A specific embodiment for implementing this system is described below.

[0141] System Configuration

[0142] The system consists of the following main components:

[0143] 1. Email summary methods:

[0144] The server parses the body of the email, extracts key information and generates a summary, using the natural language processing library spaCy and the Hugging Face Transformer model.

[0145] 2. How to convert keywords in your native language into Japanese sentences:

[0146] The server analyzes the native language keywords entered by the user and generates corresponding Japanese sentences using a predefined dictionary.

[0147] 3. A way to provide real-time summaries for efficient email review in autonomous vehicles:

[0148] The server provides real-time summaries of emails, allowing drivers and passengers to safely check their emails using their smartphones or head-mounted displays.

[0149] Program processing explanation

[0150] To summarize the email, the server first parses the body of the email using the natural language processing library spaCy to extract the email's key information, then uses the Hugging Face Transformer model to generate a summary based on the extracted information.

[0151] To convert native language keywords into Japanese sentences, the server analyzes the keywords entered by the user and generates the corresponding Japanese sentences using a predefined dictionary.

[0152] The server then sends the generated summary to a smartphone or head-mounted display as a means of providing a real-time summary for efficient email review within the autonomous vehicle, allowing the driver or passengers to safely review their email.

[0153] Examples

[0154] For example, suppose a user receives the following email while in an autonomous vehicle:

[0155] Email body:

[0156] "We have a meeting scheduled for tomorrow. Please bring the necessary documents."

[0157] The summary of this email is as follows:

[0158] summary:

[0159] "We have a meeting scheduled for tomorrow. Please bring the necessary documents."

[0160] Also, if a user enters the keywords "meeting" and "document," the system will convert this into Japanese.

[0161] Translated keywords:

[0162] "Meetings, materials"

[0163] Example prompt for a generative AI model:

[0164] Email body: "We have a meeting scheduled for tomorrow. Please bring the necessary documents."

[0165] Keywords: ["meeting", "document"]

[0166] In this way, the system will streamline email checking in autonomous vehicles, enabling users to quickly obtain important information while ensuring driving safety. In addition, by appropriately converting keywords in the user's native language into Japanese, user convenience will be improved.

[0167] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0168] Step 1:

[0169] The server receives the email body from the user. As input, the email body is provided. The server parses the email body using spaCy, a natural language processing library, and extracts key information. As output, the extracted key information is provided.

[0170] Step 2:

[0171] The server generates a summary based on the extracted key information using the Hugging Face Transformer model. As input, the key information extracted in step 1 is provided. The server inputs the prompt sentence into the summary generation model and generates a summary. As output, the generated summary is obtained.

[0172] Step 3:

[0173] A user inputs keywords in his / her native language. As input, keywords in the native language are provided. The server converts these keywords into Japanese sentences using a predefined dictionary. As output, the converted Japanese sentences are obtained.

[0174] Step 4:

[0175] The server sends the generated summary and the converted Japanese text to a smartphone or head-mounted display. As input, the summary generated in step 2 and the Japanese text converted in step 3 are provided. The server sends this information to the terminal so that the user can confirm it. As output, the summary and Japanese text are displayed on the user's terminal.

[0176] Step 5:

[0177] The user checks the summary and the Japanese text on the terminal. The summary and the Japanese text sent from the server are provided as input. The user checks this information and takes the next action if necessary. The output is the user's understanding and the next action.

[0178] Example 2

[0179] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0180] In today's business environment, it is important to quickly and accurately grasp the contents of meetings and documents. However, it is difficult to understand and summarize documents and meeting speech provided in different languages ​​in real time. It is also complex to provide appropriate information on visa applications and renewals and support the preparation of necessary documents. To solve these problems, a system that combines advanced natural language processing technology and speech recognition technology is required.

[0181] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for summarizing emails, a means for converting keywords in the native language into Japanese sentences, a means for summarizing advance materials into the native language, a means for instantly summarizing the speech of participants, a means for instantly researching difficult words, a means for informing what day it is today and what customs are there, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules, etc., and supporting the preparation of documents required for applications, a means for analyzing materials distributed in advance and generating summaries in the native language, a means for converting speech during a meeting into text using speech recognition technology and analyzing the text to generate summaries, and a means for analyzing materials and speech using a generation AI model and providing summaries in the user's native language. This makes it possible to quickly and accurately understand and summarize materials provided in different languages ​​and speech at meetings. In addition, it is possible to appropriately provide information on visa applications and renewals and support the preparation of necessary documents.

[0182] The "email summarization means" is a means for analyzing the body of an e-mail, extracting important information, and generating a concise summary.

[0183] The "means for converting keywords in the user's native language into Japanese sentences" refers to a means for converting keywords input in the user's native language into Japanese sentences.

[0184] The "means for summarizing advance materials in the user's native language" is a means for analyzing materials distributed in advance and generating a summary of the materials in the user's native language.

[0185] The "means for instantly summarizing participant speech" refers to a means for analyzing speech during a conference in real time and generating a summary thereof.

[0186] The "instant search means for difficult words" is a means for instantly searching difficult words that appear in meetings or documents, and providing their meanings and related information.

[0187] A "means of teaching what day and customs it is today" is a means of providing information about events and customs associated with a particular date.

[0188] The "Visa Application and Renewal Support Tool" is a tool that generates appropriate guidance based on the latest visa and immigration rules and supports the preparation of documents required for visa applications and renewals.

[0189] A "generative AI model" is an artificial intelligence model used for natural language processing and data analysis, such as for summarizing and translating text.

[0190] A "prompt" is an instruction entered into a generative AI model to cause it to perform a specific task.

[0191] "Voice recognition technology" is a technology that converts voice data into text data.

[0192] "Natural language processing technology" is technology for understanding, analyzing, and generating human language.

[0193] This invention is a system for quickly and accurately understanding and summarizing materials and conference speeches provided in different languages, and also includes a function for providing appropriate information on visa applications and renewals and supporting the preparation of necessary documents.

[0194] System Configuration

[0195] The system is composed of three main components: a server, a terminal, and a user. The server uses a generative AI model to analyze materials and speech and provides summaries in the user's native language. The terminal converts speech during meetings into text using speech recognition technology and sends the text to the server. Users can upload materials and summarize meeting speech in real time through the system.

[0196] Hardware and Software Used

[0197] Server: A server with high-performance computing power is required, preferably equipped with a GPU to run the generative AI model.

[0198] Terminal: A terminal with a microphone to execute the voice recognition technology and communication capabilities to send the converted text data to the server is required.

[0199] software:

[0200] Generative AI models: Use natural language processing models such as OpenAI's GPT-4(registered trademark) and Google's(registered trademark) BERT.

[0201] Speech Recognition Technology: We use voice recognition software such as Google Speech-to-Text and IBM Watson® Speech to Text.

[0202] Translation API: Use a translation API such as DeepL or Google Translate.

[0203] Program Processing

[0204] The server receives the advance materials uploaded by the user, analyzes the contents of the materials using natural language processing technology, and generates a summary. The generated summary is translated into the user's native language using a translation API and provided to the user.

[0205] The device converts speech during the meeting into text in real time using speech recognition technology and sends the text to the server. The server analyzes the received text and generates a summary. The generated summary is translated into the user's native language using a translation API and provided to the user in real time.

[0206] Examples

[0207] For example, when a user uploads English minutes, the server receives them, analyzes them, generates a summary in Japanese, and provides it to the user. In addition, the device recognizes speech spoken in English during a meeting, converts it into text, and sends it to the server. The server analyzes the text, generates a summary in Japanese, and provides it to the user in real time.

[0208] Example of prompt when summarizing minutes:

[0209] "Please summarize the following English minutes in Japanese."

[0210] Example prompts for summarizing meeting utterances:

[0211] "Please summarize the following English statements in Japanese."

[0212] The system will enable users to quickly and accurately understand and summarize materials and conference speeches provided in different languages, and will also provide appropriate information on visa applications and renewals, and assist with the preparation of necessary documents.

[0213] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0214] Step 1:

[0215] The user uploads advance materials. The user uploads materials such as meeting agendas and minutes through the system interface. The input is a document file such as PDF or Word, and the output is the material data sent to the server.

[0216] Step 2:

[0217] The server receives the materials. The server stores the uploaded materials and prepares them for analysis. The input is the material data sent by the user, and the output is the material data to be analyzed.

[0218] Step 3:

[0219] The server analyzes the document. The server uses natural language processing technology to understand the content of the document. In this case, a generative AI model (for example, GPT-4 or BERT) is used. The input is the document data to be analyzed, and the output is a summary of the document.

[0220] Step 4:

[0221] The server generates the summary. It uses a generative AI model to extract the main points of the material and create a concise summary. The input is the analyzed material data, and the output is the generated summary data.

[0222] Step 5:

[0223] The server translates the summary into the user's native language. Using a translation API (e.g., DeepL or Google Translate), the generated summary is translated into the user's native language. The input is the generated summary data, and the output is the translated summary data.

[0224] Step 6:

[0225] The server provides the translated summary to the user, who can view the summary through the system's interface. The input is the translated summary data, and the output is the summary information provided to the user.

[0226] Step 7:

[0227] The terminal recognizes speech during the meeting. A microphone installed in the meeting room picks up the speech and obtains the voice data. The input is the voice data during the meeting, and the output is the voice data sent to the voice recognition software.

[0228] Step 8:

[0229] The device converts the voice to text. It uses voice recognition software such as Google Speech-to-Text or IBM Watson Speech to Text to convert the voice data into text data. The input is voice data and the output is text data.

[0230] Step 9:

[0231] The terminal transmits the converted speech to the server. A communication protocol with low latency is used to send data to the server in real time. The input is text data, and the output is text data sent to the server.

[0232] Step 10:

[0233] The server analyzes the received text. It uses natural language processing technology to understand the content of the speech. The input is text data, and the output is the analyzed speech data.

[0234] Step 11:

[0235] The server generates a summary of the speech. It uses a generative AI model to extract the main points of the speech and create a concise summary. The input is the analyzed speech data, and the output is the generated summary data.

[0236] Step 12:

[0237] The server translates the generated summary into the user's native language using a translation API such as DeepL or Google Translate. The input is the generated summary data, and the output is the translated summary data.

[0238] Step 13:

[0239] The server provides the translated summary to the user, who can view the summary in real time. The input is the translated summary data, and the output is the summary information provided to the user.

[0240] (Application example 2)

[0241] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0242] In today's brick-and-mortar stores, it is important for store clerks to respond quickly and accurately to customer requests and questions. However, language barriers and an overload of information can make it difficult for store clerks to accurately understand what customers are saying and respond appropriately. In addition, when store clerks are required to provide multilingual support, it is difficult for them to translate and understand in real time. This can lead to lower customer satisfaction and delayed responses.

[0243] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes a means for summarizing emails, a means for converting keywords in the native language into Japanese sentences, a means for summarizing advance materials into the native language, a means for instantly summarizing the speech of participants, a means for instantly researching difficult words, a means for informing what day it is today and what customs exist, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules, etc., and for supporting the preparation of documents required for application, a means for real-time speech recognition and text conversion of customer speech, a means for analyzing the text content of the speech and generating a summary, and a means for translating the summary into multiple languages ​​and displaying it. This enables store clerks to quickly and accurately understand the content of customer speech and to instantly provide multilingual support.

[0244] The "email summarization means" is a means for analyzing the body of an e-mail, extracting important information, and generating a summary.

[0245] The "means for converting keywords in a user's native language into Japanese sentences" refers to a means for converting keywords entered in a user's native language into Japanese sentences.

[0246] The "means for summarizing advance materials into the user's native language" refers to a means for analyzing materials distributed in advance and generating a summary of the materials in the user's native language.

[0247] The "means for instantly summarizing participant speech" refers to a means for analyzing speech during a conference in real time and generating a summary thereof.

[0248] An "instant lookup tool for difficult words" is a means for instantly looking up difficult words or technical terms that appear in a conversation and providing their meaning.

[0249] A "means of informing users of today's events and customs related to the current date" is a means of informing users of today's events and customs related to the current date.

[0250] "Visa Application and Renewal Support Tool" is a tool that provides guidance necessary for visa applications and renewals based on the latest visa and immigration rules, and supports the preparation of necessary documents.

[0251] "Means for performing speech recognition on a customer's speech in real time and converting the speech into text" refers to means for performing speech recognition on a customer's speech in real time and converting the content into text.

[0252] The "means for analyzing the textualized content of an utterance and generating a summary" refers to a means for analyzing the textualized content of an utterance and generating a summary thereof.

[0253] The "means for translating and displaying a summary in multiple languages" refers to a means for translating the generated summary into multiple languages ​​and displaying it to the user.

[0254] A system for carrying out this invention has the following configuration: The server includes means for summarizing emails, means for converting keywords in a person's native language into Japanese sentences, means for summarizing advance materials into the person's native language, means for instantly summarizing speeches of participants, means for instantly researching difficult words, means for informing participants what day it is and what customs exist today, means for supporting visa applications and renewals, means for speech recognition and text conversion of customer speech in real time, means for analyzing the content of the text conversion and generating a summary, and means for translating and displaying the summary into multiple languages.

[0255] Hardware and Software Used

[0256] Hardware

[0257] Smart glasses: A device with a display to capture customer speech and display a summary.

[0258] Microphone: A voice input device for capturing customer speech.

[0259] software

[0260] Speech recognition engine: Uses Google Cloud Speech-to-Text API.

[0261] Summary generation model: Uses OpenAI GPT-4.

[0262] Translation engine: Uses Google Cloud Translation API.

[0263] Data processing and calculation

[0264] 1. Voice capture: The microphone in the smart glasses captures the customer's speech, which then captures the customer's speech as voice data.

[0265] 2. Speech recognition: The captured voice data is converted to text using the Google Cloud Speech-to-Text API, which converts the voice data into text data.

[0266] 3. Summary generation: The converted speech is analyzed using OpenAI GPT-4 to generate a summary, converting long speech into a concise summary.

[0267] 4. Translation: The summarized text is translated into the clerk’s native language using the Google Cloud Translation API, making the summary multilingual.

[0268] 5. Display: The translated summary is displayed on the smart glasses display, allowing the store attendant to quickly and accurately understand the customer's request.

[0269] Examples

[0270] For example, consider a sales associate interacting with an English-speaking customer whose native language is Japanese. If the customer says, "I am looking for a red dress in size M," the system would:

[0271] 1. The microphone in the smart glasses captures the customer's speech.

[0272] 2. Convert the audio data into text using the Google Cloud Speech-to-Text API.

[0273] 3. Analyze the transcribed speech using OpenAI GPT-4 and generate a summary.

[0274] 4. Translate the summary into Japanese using the Google Cloud Translation API.

[0275] 5. Display the translated summary “Looking for a red dress in size medium” on the smart glasses display.

[0276] Examples of prompt statements

[0277] Customer utterance: "I am looking for a red dress in size M"

[0278] Generated summary: "Looking for a red dress in size medium"

[0279] In this way, the salesperson can quickly and accurately understand the customer's request and respond appropriately.

[0280] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0281] Step 1:

[0282] The microphone in the smart glasses captures the customer's speech. The input is the customer's voice data, and the output is the captured voice data. The smart glasses record the customer's speech in real time and send the voice data to the server.

[0283] Step 2:

[0284] The server converts the captured voice data into text data using the Google Cloud Speech-to-Text API. The input is voice data and the output is text data. The server calls a speech recognition engine to analyze the voice data and generate the corresponding text.

[0285] Step 3:

[0286] The server analyzes the textual content of the utterance using OpenAI GPT-4 and generates a summary. The input is the text data and the output is the summarized text. The server uses a generative AI model to extract the important parts of the text data and create a concise summary.

[0287] Step 4:

[0288] The server translates the summarized text into the clerk's native language using the Google Cloud Translation API. The input is the summarized text and the output is the translated text. The server calls the translation engine to make the summarized text multilingual.

[0289] Step 5:

[0290] The server displays the translated summary on the display of the smart glasses. The input is the translated text and the output is the summary displayed on the display of the smart glasses. The smart glasses display the translated text received from the server on their display, allowing the store clerk to quickly understand the customer's request.

[0291] Example 3

[0292] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0293] In today's information society, users need to quickly and accurately obtain a wide variety of information. However, when faced with difficult words such as technical terms, abbreviations, and foreign languages, it is difficult to immediately understand their meaning and use. In addition, in daily life and business, it is important to appropriately obtain and respond to information about holidays, anniversaries, and visa applications, but there is no system that provides this information centrally. Therefore, there is a need for a system that can instantly look up the meaning and use of words that users do not understand and provide information about holidays, anniversaries, and visa applications.

[0294] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0295] In this invention, the server includes a means for analyzing words that the user does not understand and investigating their meanings and uses, a means for providing information on Japanese holidays, anniversaries, and seasonal events, and a means for generating a list of necessary documents based on information input by the user and instructing how to prepare each document. This allows the user to instantly understand the meanings and uses of difficult words and obtain information on holidays, anniversaries, and visa applications in a centralized manner.

[0296] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[0297] The "means for converting keywords in a user's native language into Japanese sentences" refers to a means for converting keywords entered in a user's native language into Japanese sentences.

[0298] A "means for summarizing advance materials into a user's native language" is a means for summarizing advance provided materials into a user's native language.

[0299] The "means for instantly summarizing participant utterances" refers to a means for instantly summarizing the contents of utterances made by participants in a conference or dialogue.

[0300] An "instant lookup means for difficult words" is a means for analyzing words that a user does not understand and instantly looking up their meaning and usage.

[0301] "Means of teaching what day it is and what customs it has" is a means of providing information about holidays, anniversaries, and seasonal events related to the current date.

[0302] "Means to generate appropriate guidance on visa applications and renewals and support the preparation of documents required for applications" refers to means to generate a list of documents required for visa applications and renewals based on the latest visa and immigration rules, and to provide guidance on how to prepare each document.

[0303] "A means for analyzing words that a user does not understand and investigating their meaning and use" refers to a means for analyzing difficult words entered by a user and investigating their meaning and use from information sources on the Internet.

[0304] "Means for providing information about Japanese holidays, anniversaries, and seasonal events" refers to means for retrieving information about Japanese holidays, anniversaries, and seasonal events from a database and providing it to a user.

[0305] "A means for generating a list of required documents based on information input by the user and providing instructions on how to prepare each document" refers to a means for generating a list of required documents based on information input by the user, such as the type of visa, period of stay, and purpose of the application, and for providing instructions on how to prepare each document and where to submit it.

[0306] This invention is a system that analyzes words and technical terms that users do not understand, and investigates and provides their meaning and usage. It also provides information on Japanese holidays, anniversaries, and seasonal events, and generates guidance on visa applications and renewals.

[0307] System Configuration

[0308] Hardware and Software

[0309] Server: A central processing unit for data analysis and information provision. It uses programming languages ​​such as Python and Java (registered trademark), and MySQL (registered trademark) for the database.

[0310] Terminal: A device on which the user enters information and displays the results. This includes PCs, smartphones, tablets, etc. HTML, CSS, and JavaScript (registered trademark) are used for the front end.

[0311] Internet: A network for data communication between servers and terminals.

[0312] Program Processing

[0313] Word analysis and research

[0314] 1. The user types a word that the user does not understand into a terminal input field.

[0315] 2. The device sends the input words to the server, specifically by sending the data using an HTTP request.

[0316] 3. The server analyzes the received words. It uses natural language processing (NLP) techniques, such as the Python NLTK library, to analyze them. It then calls the Wikipedia API or the API of a technical dictionary to investigate the meaning of the words.

[0317] 4. The server sends the investigation results in JSON format to the terminal, including, for example, "AI is an abbreviation for Artificial Intelligence, a technology that allows machines to mimic human intelligence."

[0318] 5. The device parses the received JSON data and displays it in a user-friendly format, using HTML and CSS to display the results on the screen.

[0319] Providing information on holidays and anniversaries

[0320] 1. The user types "What day is it today?" into the input field on the terminal.

[0321] 2. The device sends the input to the server, specifically by sending the data using an HTTP request.

[0322] 3. The server looks up a database of Japanese holidays and anniversaries (for example, a MySQL database) and uses an SQL query to find the holidays and anniversaries that fall on today's date.

[0323] 4. The server sends the acquired information in JSON format to the terminal, including, for example, "Today is Respect for the Aged Day. Respect for the Aged Day is a day to respect the elderly and celebrate longevity."

[0324] 5. The device parses the received JSON data and displays it in a user-friendly format, using HTML and CSS to display the results on the screen.

[0325] Visa Application and Renewal Guidance

[0326] 1. The user enters "Tourist visa, 3 months, tourist purpose" into the input field on the terminal.

[0327] 2. The device sends the input to the server, specifically by sending the data using an HTTP request.

[0328] 3. The server generates a list of required documents based on the input information. For example, it could use Python logic to generate a list like "To apply for a tourist visa, the following documents are required: application form, copy of passport, proof of place of stay."

[0329] 4. The server generates detailed guidance on how to prepare each document and where to submit it, for example, "Please download the application form from the official website and fill in the necessary information. Please make a copy of every page of your passport."

[0330] 5. The server sends the generated guidance in JSON format to the device.

[0331] 6. The device parses the received JSON data and displays it in a user-friendly format, using HTML and CSS to display the results on the screen.

[0332] Examples of concrete examples and prompts

[0333] Examples of word analysis and research

[0334] A user types, "What is AI?"

[0335] The device sends "AI" to the server.

[0336] The server investigates the meaning of "AI" and sends the result to the terminal, stating that "AI is an abbreviation for Artificial Intelligence, a technology that allows machines to imitate human intelligence."

[0337] The terminal displays the results to the user.

[0338] Examples of providing information about holidays and anniversaries

[0339] The user types, "What day is it today?"

[0340] The terminal sends input to the server.

[0341] The server sends the result to the terminal, stating, "Today is Respect for the Aged Day. Respect for the Aged Day is a day to respect the elderly and celebrate longevity."

[0342] The terminal displays the results to the user.

[0343] Examples of guidance for visa application and renewal

[0344] The user enters "Tourist visa, 3 months, tourist purpose."

[0345] The terminal sends input to the server.

[0346] The server sends the result to the terminal as "The following documents are required to apply for a tourist visa: application form, copy of passport, and proof of place of stay."

[0347] The terminal displays the results to the user.

[0348] Examples of prompt statements

[0349] "What is AI?"

[0350] "Tell me what day it is."

[0351] Please tell me what documents I need to apply for a tourist visa.

[0352] In this way, the system searches for necessary information based on the user's input and provides it in an easy-to-understand manner. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0353] Word analysis and research

[0354] Step 1:

[0355] The user types a word that the device does not understand into an input field.

[0356] Input: What the user types (e.g. "What is AI?")

[0357] Output: Words typed into the terminal

[0358] Step 2:

[0359] The device sends the input words to the server, specifically by sending the data using an HTTP request.

[0360] Input: Words typed into the terminal

[0361] Output: The HTTP request sent to the server.

[0362] Step 3:

[0363] The server analyzes the received words using natural language processing (NLP) techniques, such as the Python NLTK library.

[0364] Input: The word sent to the server

[0365] Output: Meaning and usage of analyzed words

[0366] Step 4:

[0367] The server calls the Wikipedia API or a technical dictionary API to look up the meaning of words.

[0368] Input: Parsed words

[0369] Output: Research results (e.g. "AI stands for Artificial Intelligence, a technology that allows machines to mimic human intelligence")

[0370] Step 5:

[0371] The server sends the investigation results to the terminal in JSON format.

[0372] Input: Survey results

[0373] Output: JSON data sent to the terminal

[0374] Step 6:

[0375] The device parses the received JSON data and displays it in a user-friendly format, using HTML and CSS to display the results on the screen.

[0376] Input: JSON data sent to the terminal

[0377] Output: Findings displayed to the user

[0378] Providing information on holidays and anniversaries

[0379] Step 1:

[0380] The user types "What day is it today?" into the device's input field.

[0381] Input: The question the user types in (e.g. "What day is it today?")

[0382] Output: Question typed into the terminal

[0383] Step 2:

[0384] The device sends input to the server, specifically by sending data using an HTTP request.

[0385] Input: Question typed into the terminal

[0386] Output: The HTTP request sent to the server.

[0387] Step 3:

[0388] The server looks up a database of Japanese holidays and anniversaries (for example, a MySQL database) and uses an SQL query to find the holidays and anniversaries that fall on today's date.

[0389] Input: The question sent to the server

[0390] Output: Holiday and anniversary information retrieved from the database

[0391] Step 4:

[0392] The server sends the acquired information to the terminal in JSON format.

[0393] Input: Information retrieved from a database

[0394] Output: JSON data sent to the terminal

[0395] Step 5:

[0396] The device parses the received JSON data and displays it in a user-friendly format, using HTML and CSS to display the results on the screen.

[0397] Input: JSON data sent to the terminal

[0398] Output: Holiday and anniversary information displayed to the user

[0399] Visa Application and Renewal Guidance

[0400] Step 1:

[0401] The user types "Tourist visa, 3 months, tourist purpose" into the device's input field.

[0402] Input: Visa information entered by the user (e.g. "Tourist Visa, 3 months, tourist purpose")

[0403] Output: Visa information entered into the terminal

[0404] Step 2:

[0405] The device sends input to the server, specifically by sending data using an HTTP request.

[0406] Input: Visa information entered into the terminal

[0407] Output: The HTTP request sent to the server.

[0408] Step 3:

[0409] The server generates a list of required documents based on the input information, for example using Python logic to generate a list like "To apply for a tourist visa, the following documents are required: application form, copy of passport, proof of place of stay".

[0410] Input: Visa information sent to the server

[0411] Output: A list of generated documents

[0412] Step 4:

[0413] The server generates detailed guidance on how to prepare each document and where to submit it, for example, "Please download the application form from the official website and fill in the necessary information. Please make a copy of every page of your passport."

[0414] Input: A list of generated documents

[0415] Output: Detailed guidance

[0416] Step 5:

[0417] The server generates guidance and sends it to the terminal in JSON format.

[0418] Input:Detailed Guidance

[0419] Output: JSON data sent to the terminal

[0420] Step 6:

[0421] The device parses the received JSON data and displays it in a user-friendly format, using HTML and CSS to display the results on the screen.

[0422] Input: JSON data sent to the terminal

[0423] Output: Guidance displayed to the user

[0424] (Application example 3)

[0425] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0426] Today's consumers want to instantly obtain detailed information about products and special event information to improve their shopping experience in physical stores. However, it is difficult to immediately understand product descriptions that contain many technical terms and abbreviations, or event information held in stores. In addition, foreign tourists need to quickly and accurately obtain information about visa applications and renewals. A system that can solve these problems is needed.

[0427] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0428] In this invention, the server includes a means for summarizing emails, a means for converting keywords in the native language into Japanese sentences, a means for summarizing advance materials into the native language, a means for instantly summarizing speeches of participants, a means for instantly researching difficult words, a means for informing a participant of what day it is and what customs are in place today, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules, and for supporting the preparation of documents required for applications, a means for scanning technical terms and abbreviations contained in product packages and descriptions with a camera and instantly researching and displaying their meanings and uses, a means for providing information on today's holidays and anniversaries, special events and sales taking place in stores, and a means for providing guidance on visa applications and renewals for foreign tourists, and for instructing on a list of necessary documents and how to prepare them. This allows consumers to improve their shopping experience in physical stores, and allows foreign tourists to quickly and accurately obtain information on visa applications and renewals.

[0429] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[0430] The "means for converting keywords in a user's native language into Japanese sentences" refers to a means for converting keywords entered in a user's native language into Japanese sentences.

[0431] A "means for summarizing advance materials into a user's native language" is a means for summarizing advance provided materials into a user's native language.

[0432] The "means for instantly summarizing participant speech" refers to a means for summarizing participant speech in real time during a meeting or discussion.

[0433] An "instant lookup tool for difficult words" is a tool that allows users to instantly look up technical terms or abbreviations that they do not understand and provide their meanings and uses.

[0434] "Means of teaching what day it is and what customs exist" refers to a means of providing information about what day it is and the customs and events that take place on that day.

[0435] "A means of generating appropriate guidance regarding visa applications and renewals in accordance with the latest visa and immigration rules, etc., and supporting the preparation of documents required for applications" refers to a means of providing guidance regarding visa applications and renewals based on the latest visa and immigration rules, and supporting the preparation of necessary documents.

[0436] "A means for scanning technical terms and abbreviations contained in product packaging or descriptions with a camera, and instantly researching and displaying their meanings and uses" refers to a means for scanning technical terms and abbreviations contained in product packaging or descriptions with a camera, and instantly researching and displaying their meanings and uses to the user.

[0437] "Means for providing information about today's holidays and anniversaries, and about special events and sales taking place at stores" refers to means for providing users with information about today's holidays and anniversaries, and about special events and sales taking place at stores.

[0438] "Means of providing guidance to foreign tourists regarding visa applications and renewals, and instructing them on the list of required documents and how to prepare them" means means of providing guidance to foreign tourists regarding visa applications and renewals, and instructing them on the list of required documents and how to prepare them.

[0439] The system for implementing this invention uses the following hardware and software. The hardware uses a smartphone or smart glasses. The software uses a camera API, a natural language processing library (e.g., spaCy), a database (e.g., Firebase), and a generative AI model (e.g., GPT-4).

[0440] System Configuration

[0441] 1. Email summary methods:

[0442] The server analyzes the body of the email, extracts important information, and generates a summary, which allows the user to quickly grasp the important information.

[0443] 2. How to convert keywords in your native language into Japanese sentences:

[0444] The server converts keywords entered in the user's native language into Japanese sentences, making it easier for foreign users to understand information in Japanese.

[0445] 3. How to summarize the advance materials in your native language:

[0446] The server summarizes the pre-provided materials in the user's native language, allowing the user to quickly understand the contents of the materials.

[0447] 4. Means of immediate summary of participants' speech:

[0448] The server summarizes the speech of participants during meetings and discussions in real time, allowing users to efficiently grasp the contents of the meetings.

[0449] 5. Quick lookup for difficult words:

[0450] The server instantly searches for technical terms and abbreviations that the user does not understand and provides their meaning and usage, thereby enabling the user to easily understand technical information.

[0451] 6. Ways to teach what day it is and what customs it has:

[0452] The server provides information about what day it is, customs and events taking place on that day, etc. This allows users to obtain information that is useful in their daily lives.

[0453] 7. To generate appropriate guidance on visa application and renewal in line with the latest visa and immigration regulations, and to assist in preparing the necessary documents for application:

[0454] The server provides guidance on visa applications and renewals based on the latest visa and immigration regulations, and assists in preparing the necessary documents, allowing foreign users to smoothly apply for and renew their visas.

[0455] 8. A method for scanning technical terms and abbreviations in product packaging and descriptions with a camera and instantly researching and displaying their meaning and usage:

[0456] The device uses a camera to scan technical terms and abbreviations contained in product packaging and descriptions, and instantly researches their meanings and uses and displays them to the user, allowing the user to instantly obtain detailed information about the product.

[0457] 9. Ways to share information about current holidays, anniversaries, special events and sales happening in your store:

[0458] The server provides the user with information about today's holidays and anniversaries, as well as special events and sales taking place in stores, making it easier for the user to plan their shopping.

[0459] 10. Providing guidance to foreign tourists on visa applications and renewals, listing the necessary documents and providing instructions on how to complete them:

[0460] The server provides guidance to foreign tourists on visa applications and renewals, including a list of required documents and instructions on how to prepare them, allowing foreign tourists to efficiently apply for and renew visas.

[0461] Examples of concrete examples and prompts

[0462] Examples

[0463] Instant terminology research:

[0464] When a user uses the smart glasses to scan a product package, the term "gluten-free" appears and its meaning and uses are instantly explained.

[0465] Today's event information:

[0466] When a user opens the app, they are shown the information, "Today is Halloween. Special sales are happening at stores near you."

[0467] Visa Application Guidance:

[0468] When a foreign tourist enters "tourist visa, 3 months, tourism purpose" into the app, a list of required documents and instructions on how to prepare them are displayed.

[0469] Examples of prompt statements

[0470] Instant terminology research:

[0471] "Research the meaning and use of this term: gluten-free."

[0472] Today's event information:

[0473] "Please let me know today's holidays, anniversaries, and store events."

[0474] Visa Application Guidance:

[0475] "Please tell me the documents required to apply for a tourist visa and how to fill them out. I will be staying for 3 months and my purpose is tourism."

[0476] In this way, the system can provide users with a variety of information instantly, improving their shopping experience in brick-and-mortar stores, and providing foreign tourists with quick and accurate information on visa applications and renewals.

[0477] The flow of the specific process in the application example 3 will be described with reference to FIG.

[0478] Step 1:

[0479] A user scans a product package using a smartphone or smart glasses.

[0480] Input: Product packaging image

[0481] Output: The extracted text.

[0482] Specific operation: The device's camera API is used to obtain an image of the product packaging, and text is extracted using OCR (optical character recognition) technology.

[0483] Step 2:

[0484] The server analyzes the extracted text to identify technical terms and abbreviations.

[0485] Input: Extracted text

[0486] Output: Identified terms and abbreviations

[0487] What happens: The server uses a natural language processing library (e.g. spaCy) to analyze the text and identify jargon and abbreviations.

[0488] Step 3:

[0489] The server investigates the meaning and usage of specified technical terms and abbreviations from Internet sources.

[0490] Input: Identified terminology or abbreviation

[0491] Output: Meaning and usage of technical terms and abbreviations

[0492] Specific Behavior: The server uses Internet resources (e.g., Wikipedia API) to look up the meaning and usage of identified terms and abbreviations.

[0493] Step 4:

[0494] The server displays the results of the investigation to the user.

[0495] Input: Meaning and use of technical terms and abbreviations

[0496] Output: Information displayed to the user

[0497] Specific operation: The server transmits the investigation result to the terminal, and the terminal displays the investigation result to the user.

[0498] Step 5:

[0499] The server provides information about today's holidays and anniversaries, as well as special events and sales happening in stores.

[0500] Input: Current date and user location

[0501] Output: Holidays, anniversaries, and event information

[0502] Specific operation: The server retrieves information about today's holidays, anniversaries, and store events from a database (e.g. Firebase) and provides relevant information based on the user's location.

[0503] Step 6:

[0504] The server provides guidance to foreign tourists on visa applications and renewals.

[0505] Input: User's visa type, duration of stay, purpose of application

[0506] Output: List of required documents and how to create them

[0507] How it works: The server uses a generative AI model (e.g. GPT-4) to generate a list of required documents and instructions on how to create them based on the user's input, and provides them to the user.

[0508] Step 7:

[0509] The user acts on the information provided.

[0510] Input: Information provided by the server

[0511] Output: User actions

[0512] Specific operations: Based on the information provided by the server, the user purchases products, participates in events, applies for visas, etc.

[0513] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0514] "Example 1"

[0515] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system analyzes emotions from a user's text input or voice input. Specifically, when a user composes an email, the system analyzes the user's emotional state from the content of the email. For example, if the user uses the expression "I'm in trouble," the system analyzes that the user is facing a difficult situation.

[0516] "Example 2"

[0517] Based on the analyzed emotions, the system can tailor its responses and services to the user. For example, if the system detects that the user is facing a difficult situation, it can provide more specific assistance to the user, such as providing detailed guidance on the visa application process or providing additional information the user may need.

[0518] "Example 3"

[0519] Furthermore, the emotion engine works in conjunction with other functions of the system. For example, the email summary function adjusts the content of the summary by taking into account the user's emotional state analyzed by the emotion engine. For example, if the user expresses anger, the summary function will highlight that information. This allows the user to understand their own emotional state more clearly and take appropriate action.

[0520] The process flow of each embodiment will be described below.

[0521] "Example 1"

[0522] Step 1: The user enters text or voice input into the system.

[0523] Step 2: The sentiment engine analyzes the user's input for sentiment, taking into account word choice, context, tone of voice, etc.

[0524] Step 3: The analysis results are fed back to other functions of the system, allowing it to respond according to the user's emotional state.

[0525] "Example 2"

[0526] Step 1: The emotion engine analyzes the user's emotions.

[0527] Step 2: Based on the analysis results, the system tailors its response and services to the user.

[0528] Step 3: For example, if the system analyzes that the user is facing a difficult situation, it will provide more specific assistance to the user.

[0529] "Example 3"

[0530] Step 1: The emotion engine analyzes the user's emotions.

[0531] Step 2: The analysis results are fed back to other functions of the system, such as the email digest function.

[0532] Step 3: The email summary feature adjusts the summary content taking into account the user's emotional state as analyzed by the sentiment engine.

[0533] Example 1

[0534] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0535] In today's business environment, there is a demand for efficient processing of multiple tasks, such as summarizing emails, handling multiple languages, and supporting visa applications. However, processing these tasks individually requires a lot of time and effort, which places a heavy burden on users. It is also important to understand the user's situation through sentiment analysis and respond appropriately, but current systems do not integrate these functions, making it difficult to respond efficiently.

[0536] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0537] In this invention, the server includes a means for summarizing emails, a means for converting keywords in the user's native language into Japanese sentences, a means for summarizing advance materials into the user's native language, a means for instantly summarizing speeches of participants, a means for instantly researching difficult words, a means for informing the user of what day it is and what customs exist, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules and for supporting the preparation of documents required for applications, a means for analyzing emotions from the user's text input or voice input, and a means for generating Japanese sentences using a generative AI model. This makes it possible to efficiently process multiple tasks, reduce the user's burden, and enable appropriate responses.

[0538] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[0539] The "means for converting keywords in one's native language into sentences in Japanese" refers to a means for analyzing keywords in one's native language input by a user and generating corresponding sentences in Japanese.

[0540] The "means for summarizing advance materials into the native language" refers to a means for analyzing materials provided in advance and generating a summary in the native language.

[0541] The "means for instantly summarizing participant speech" refers to a means for summarizing in real time what is said by participants in a meeting or event.

[0542] An "instant lookup tool for difficult words" is a tool that enables users to instantly look up words or technical terms that are difficult for them to understand and provide their meanings and explanations.

[0543] A "means of teaching what day and customs it is today" is a means of providing information about events and customs associated with a particular date.

[0544] The "Visa Application and Renewal Support Tool" is a tool that generates guidance necessary for visa applications and renewals based on the latest visa and immigration rules, and supports the preparation of documents necessary for applications.

[0545] An "emotion analysis means" is a means for analyzing emotions from a user's text input or voice input and identifying the user's emotional state.

[0546] A "means for generating Japanese sentences using a generative AI model" is a means for generating Japanese sentences based on input data using a generative AI model.

[0547] This invention is a system including a means for summarizing emails, a means for converting keywords in one's native language into Japanese sentences, a means for summarizing advance materials into one's native language, a means for instantly summarizing speech by participants, a means for instantly researching difficult words, a means for informing participants what day it is and what customs exist, a means for supporting visa applications and renewals, a means for sentiment analysis, and a means for generating Japanese sentences using a generative AI model.

[0548] Email summary method

[0549] The server receives the email body and analyzes it using a natural language processing library (e.g. NLTK, spaCy). It extracts important keywords and phrases from the analyzed text and generates a summary. The summary is sent to the terminal and displayed to the user.

[0550] Examples:

[0551] A user sends an email saying, "I am sending you the minutes of the meeting. Key points are listed below."

[0552] The server extracts the keywords "meeting," "minutes," and "important points" and generates a summary sentence such as "We will send you the important points of the meeting minutes."

[0553] The terminal displays the summary to the user.

[0554] A means of converting keywords in your native language into Japanese sentences

[0555] The user enters a keyword in their native language into the device, which then sends the keyword to the server. The server uses a generative AI model (e.g. GPT-3) to analyze the keyword and generate a corresponding Japanese sentence. The generated Japanese sentence is sent to the device and displayed to the user.

[0556] Examples:

[0557] A user enters the keyword "meeting."

[0558] The server generates the Japanese sentence "conference" and sends it to the terminal.

[0559] The terminal displays Japanese text to the user.

[0560] Emotion analysis means

[0561] The user inputs text or voice into the device, and the device sends the input data to the server. The server uses an emotion analysis engine (e.g. IBM Watson, Microsoft® Azure® Text Analytics) to analyze the user's emotions and identify their emotional state. The analysis results are sent to the device and displayed to the user.

[0562] Examples:

[0563] The user enters the text "I'm having trouble."

[0564] The server analyzes the expression "I'm in trouble" and determines that the user is facing a difficult situation.

[0565] The terminal displays the analysis results to the user.

[0566] Examples of prompt statements

[0567] Email summary prompt:

[0568] Enter the body of your email and we'll extract the key information and generate a summary.

[0569] Prompt sentence for converting keywords in your native language into Japanese sentences:

[0570] Please enter a keyword in your native language and we will generate the corresponding Japanese sentence.

[0571] Emotion engine prompt:

[0572] Enter text or voice and we'll analyze the sentiment from your input.

[0573] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0574] Email summary method

[0575] Step 1:

[0576] A user sends an email.

[0577] Input: Email body

[0578] Output: Emails stored on the mail server

[0579] Specific behavior: A user composes an email using an email client and presses the send button.

[0580] Step 2:

[0581] The server receives the email from the mail server.

[0582] Input: Emails stored on the mail server

[0583] Output: The email body forwarded to the server

[0584] Specific operation: The server periodically checks the mail server and receives new mail.

[0585] Step 3:

[0586] The server tokenizes the email body using a natural language processing library (e.g. NLTK, spaCy).

[0587] Input: Email body

[0588] Output: A tokenized word list

[0589] Specific operation: The server analyzes the email body and splits it into words.

[0590] Step 4:

[0591] The server tags the tokenized words with parts of speech and extracts important keywords and phrases.

[0592] Input: A tokenized word list

[0593] Output: A list of important keywords and phrases

[0594] What it does: The server tags each word with a part-of-speech tag and applies an algorithm to rate its importance.

[0595] Step 5:

[0596] A summary is generated based on the keywords and phrases extracted by the server.

[0597] Input: A list of important keywords and phrases

[0598] Output: Summary

[0599] What it does: The server uses a summary generation algorithm to create a short sentence containing the key information.

[0600] Step 6:

[0601] The server transmits the generated summary to the terminal.

[0602] Input: Abstract

[0603] Output: Summary sent to terminal

[0604] Specific operation: The server transmits the summary text to the terminal via the network.

[0605] Step 7:

[0606] The terminal displays the summary to the user.

[0607] Input: Abstract

[0608] Output: Summary displayed to the user

[0609] Specific operation: The terminal displays the summary on the display.

[0610] A means of converting keywords in your native language into Japanese sentences

[0611] Step 1:

[0612] A user inputs a keyword in his / her native language into a terminal.

[0613] Input: Keywords in your native language

[0614] Output: Keywords typed into the terminal

[0615] Specific Action: A user enters a keyword using a keyboard or voice input.

[0616] Step 2:

[0617] The terminal transmits the entered keyword to the server.

[0618] Input: Keywords in your native language

[0619] Output: Keywords sent to the server

[0620] Specific operation: The terminal sends the keyword to the server via the network.

[0621] Step 3:

[0622] The server analyzes the keywords and generates corresponding Japanese sentences.

[0623] Input: Keywords in your native language

[0624] Output: Japanese text

[0625] Specific operation: The server uses a generative AI model (e.g. GPT-3) to generate Japanese sentences based on keywords.

[0626] Step 4:

[0627] The Japanese text generated by the server is sent to the terminal.

[0628] Input: Japanese text

[0629] Output: Japanese text sent to the terminal

[0630] Specific operation: The server sends Japanese text to the terminal via the network.

[0631] Step 5:

[0632] The terminal displays Japanese text to the user.

[0633] Input: Japanese text

[0634] Output: Japanese text displayed to the user

[0635] Specific operation: The device displays Japanese text on the display.

[0636] Emotion analysis means

[0637] Step 1:

[0638] The user inputs text or voice into the terminal.

[0639] Input: Text or audio data

[0640] Output: Text or audio data entered into the device

[0641] Specific action: A user inputs text or voice using a keyboard or microphone.

[0642] Step 2:

[0643] The terminal transmits the entered text and voice data to the server.

[0644] Input: Text or audio data

[0645] Output: The text or audio data sent to the server.

[0646] Specific operation: The device sends text and voice data to the server via the network.

[0647] Step 3:

[0648] The server analyzes the user's emotions using a sentiment analysis engine (e.g., IBM Watson, Microsoft Azure Text Analytics).

[0649] Input: Text or audio data

[0650] Output: Emotion analysis results

[0651] What happens: The server uses a sentiment analysis engine to identify sentiment from the input data.

[0652] Step 4:

[0653] The server sends the analysis results to the terminal.

[0654] Input: Sentiment analysis results

[0655] Output: Analysis results sent to the device

[0656] Specific operation: The server transmits the analysis results to the terminal via the network.

[0657] Step 5:

[0658] The terminal displays the analysis results to the user.

[0659] Input: Sentiment analysis results

[0660] Output: Analysis results displayed to the user

[0661] Specific operation: The device displays the analysis results on the display.

[0662] (Application example 1)

[0663] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0664] In today's information society, users receive large amounts of emails and messages and are required to manage them efficiently. It is also important to provide appropriate content according to the user's emotional state. However, with conventional systems, it was difficult to perform email summarization, emotional analysis, and content recommendation according to emotions in a unified manner. This often caused users to feel stressed by information overload, making it difficult to manage information efficiently.

[0665] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes an email summarizing means, a means for converting keywords in the native language into Japanese sentences, an emotion analyzing means, and a means for recommending content according to the emotional state. This allows the user to efficiently summarize received emails and messages and receive appropriate content according to the emotional state.

[0666] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[0667] The "means for converting keywords in one's native language into sentences in Japanese" refers to a means for analyzing keywords in one's native language input by a user and generating corresponding sentences in Japanese.

[0668] The "means for summarizing advance materials into the native language" refers to a means for analyzing materials provided in advance and generating a summary in the native language.

[0669] The "means for instantly summarizing participant speech" refers to a means for summarizing in real time what is said by participants in a meeting or event.

[0670] An "instant lookup tool for difficult words" is a tool that allows users to instantly look up words or technical terms that are difficult for them to understand and provide their meanings.

[0671] A "means of telling what day it is and what customs it has" is a means of providing information about events and customs related to the current date.

[0672] "Means of generating appropriate guidance regarding visa applications and renewals and supporting the preparation of documents required for applications" refers to means of providing guidance necessary for visa applications and renewals and supporting the preparation of necessary documents based on the latest visa and immigration rules.

[0673] An "emotion analysis means" is a means for analyzing emotions from a user's text input or voice input.

[0674] The "means for recommending content according to an emotional state" refers to a means for recommending appropriate content based on an analyzed emotional state of a user.

[0675] The following system configuration will be described as an embodiment of the present invention.

[0676] System Configuration

[0677] This system consists of a server and a user terminal. The server includes a means for summarizing emails, a means for converting keywords in the user's native language into Japanese sentences, a means for sentiment analysis, and a means for recommending content according to the user's emotional state. The user terminal is a device such as a smartphone, tablet, or PC, and provides an interface for users to receive emails and messages and use various functions of the system.

[0678] Hardware and Software Used

[0679] Hardware: Smartphones, tablets, computers

[0680] Software: Python, OpenAI API, TextBlob

[0681] Data processing and calculation

[0682] 1. Email summary methods:

[0683] The server analyzes the text of the email received by the user, extracts important information, and generates a summary. For this process, it uses OpenAI's GPT-3 model. Specifically, it inputs the text of the email as a prompt and generates a summary.

[0684] 2. How to convert keywords in your native language into Japanese sentences:

[0685] The server analyzes the native language keywords entered by the user and generates corresponding Japanese sentences. For this process, OpenAI's GPT-3 model is used. The keyword is entered as a prompt and Japanese sentences are generated.

[0686] 3. Emotion analysis means:

[0687] The server analyzes the sentiment from the user's text and voice input using the TextBlob library. It calculates the optimism (positive / negative) and subjectivity of the text.

[0688] 4. How to recommend content based on your emotional state:

[0689] The server recommends appropriate content based on the analyzed emotional state of the user, for example, recommending relaxing content if the user shows negative emotions.

[0690] Examples

[0691] Email summary: The email body "Please prepare for tomorrow's meeting" is summarized to "Please prepare for the meeting."

[0692] Sentiment analysis: Analyze negative emotions from the text "I'm in trouble" and recommend content to help the user relax.

[0693] Keyword conversion: Convert the keyword "meeting" to "I have a meeting."

[0694] Examples of prompt statements

[0695] Email Summary: "Please summarize the email body below:\n\n{email body}"

[0696] Keyword conversion: "Convert the following keyword into a Japanese sentence: {keyword}"

[0697] In this way, users can efficiently manage their received emails and messages and receive appropriate content according to their emotional state.

[0698] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0699] Step 1:

[0700] A user receives an email on a terminal. The terminal sends the body of the received email to a server. The input is the email body, and the output is sending the email body to the server.

[0701] Step 2:

[0702] The server analyzes the received email body, extracts important information, and generates a summary. Specifically, it uses OpenAI's GPT-3 model to generate a summary sentence by inputting the email body as a prompt. The input is the email body, and the output is the summary sentence.

[0703] Step 3:

[0704] The server sends the generated summary to the terminal, which displays the summary to the user. The input is the summary, and the output is the display of the summary to the user.

[0705] Step 4:

[0706] A user inputs a keyword in his / her native language into a terminal. The terminal transmits the input keyword to a server. The input is the keyword in the native language, and the output is the transmission of the keyword to the server.

[0707] Step 5:

[0708] The server analyzes the received keywords and generates corresponding Japanese sentences. Specifically, it uses OpenAI's GPT-3 model to input the keywords as prompts and generate Japanese sentences. The input is the native language keyword, and the output is the Japanese sentence.

[0709] Step 6:

[0710] The server sends the generated Japanese text to the terminal. The terminal displays the Japanese text to the user. The input is the Japanese text, and the output is the display of the Japanese text to the user.

[0711] Step 7:

[0712] A user inputs text or voice into a terminal. The terminal sends the input text or voice data to a server. The input is text or voice data, and the output is the transmission of data to the server.

[0713] Step 8:

[0714] The server analyzes the sentiment from the received text or audio data. Specifically, it uses the TextBlob library to calculate the optimism (positive / negative) and subjectivity of the text. The input is the text or audio data, and the output is the sentiment analysis result.

[0715] Step 9:

[0716] The server recommends appropriate content based on the analyzed emotional state. For example, if the user shows negative emotions, it recommends relaxing content. The input is the emotion analysis result, and the output is the recommended content.

[0717] Step 10:

[0718] The server sends the recommended content to the terminal. The terminal displays the recommended content to the user. The input is the recommended content, and the output is the display of the content to the user.

[0719] Example 2

[0720] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0721] In today's business environment, there is a need to quickly and accurately grasp the content of meetings and documents. However, understanding the content of documents and meetings provided in different languages ​​is a major challenge, especially for multinational companies. It is also important to provide appropriate responses and services based on the user's emotions, but there are only a limited number of systems that can do this automatically. Furthermore, the procedures for visa applications and renewals are complicated, making it difficult for users to receive appropriate guidance.

[0722] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for summarizing emails, a means for converting keywords in the native language into Japanese sentences, a means for summarizing advance materials into the native language, a means for instantly summarizing the speech of participants, a means for instantly researching difficult words, a means for informing what day it is today and what customs exist, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules, etc., and for supporting the preparation of documents required for application, and a means for adjusting responses and services to users based on emotion analysis. This enables users to quickly understand materials and meeting contents in different languages ​​and receive appropriate guidance and services.

[0723] The "email summarization means" is a means for analyzing the body of an e-mail, extracting important information, and generating a concise summary.

[0724] The "means for converting keywords in a user's native language into Japanese sentences" refers to a means for converting keywords entered in a user's native language into Japanese sentences.

[0725] The "means for summarizing advance materials into the user's native language" refers to a means for analyzing materials distributed in advance and generating a summary of the materials in the user's native language.

[0726] The "means for instantly summarizing participant speech" refers to a means for analyzing speech during a conference in real time and generating a summary thereof.

[0727] An "instant lookup tool for difficult words" is a tool that instantly looks up difficult words and technical terms and provides their meanings and explanations.

[0728] A "means of teaching what day and customs it is today" is a means of providing information about events and customs associated with a particular date.

[0729] "Means of generating appropriate guidance regarding visa applications and renewals and supporting the preparation of documents required for applications" means means of providing appropriate guidance regarding visa applications and renewals and supporting the preparation of necessary documents based on the latest visa and immigration rules.

[0730] "Means for adjusting responses and services to users based on emotion analysis" refers to means for analyzing the emotions of users and providing appropriate responses and services based on the results.

[0731] The present invention provides a system that enables a user to quickly understand materials and meeting contents in different languages ​​and receive appropriate guidance and services. A specific embodiment of this system will be described below.

[0732] 1. Email Summary

[0733] The server analyzes the body of the email, extracts important information, and generates a concise summary. The software used includes APIs that provide natural language processing techniques. For example, a natural language processing API is used to analyze the content of the email and extract key points.

[0734] Examples:

[0735] The server analyzes the long email received by the user and generates a summary. For example, "Please summarize the following email."

[0736] 2. A way to convert keywords in your native language into Japanese sentences

[0737] The server converts the keywords entered in the user's native language into Japanese text. The software used includes a translation API. For example, the server converts the keywords into Japanese text using a translation API.

[0738] Examples:

[0739] When the user enters "meeting" in English, the server converts it to "会議" in Japanese. For example, "Please convert the following keywords into Japanese."

[0740] 3. Summary means for pre-distributed materials into the native language

[0741] The server analyzes the pre-distributed materials and generates a summary in the user's native language. The software used includes natural language processing technology and translation APIs. For example, use the natural language processing API to analyze the content of the materials, extract important points, and use the translation API to translate the summary into the native language.

[0742] Specific example:

[0743] The user uploads the English meeting minutes, and the server analyzes the minutes and generates a Japanese summary. For example, "Please summarize the following English meeting minutes in Japanese."

[0744] 4. Real-time summary means for participants' speeches

[0745] The server analyzes the speeches during the meeting in real time and generates a summary. The hardware used includes a microphone, and the software includes a speech recognition API and a natural language processing API. For example, use the speech recognition API to convert the speech into text, and use the natural language processing API to generate a summary.

[0746] Specific example:

[0747] Convert the English speeches during the meeting into text in real time, analyze the text, and generate a Japanese summary. For example, "Please convert the following voice data into text and summarize it in Japanese."

[0748] 5. Real-time investigation means for difficult words

[0749] The server instantly searches difficult words and technical terms and provides their meanings and explanations. The software used includes a dictionary API and a knowledge base. For example, the dictionary API is used to get the meaning of difficult words.

[0750] Examples:

[0751] If a user types in the word "computer vision," the server looks up the meaning and provides an explanation, e.g., "Please look up the meaning of the following words."

[0752] 6. A way to tell people what day it is and what customs apply

[0753] The server provides information about events and customs associated with a particular date. The software used may include a calendar API and an event database. For example, the calendar API may be used to retrieve event information for a particular date.

[0754] Examples:

[0755] When a user asks "What day is it today?", the server provides the events and customs of that day. For example, "Tell me what day it is today."

[0756] 7. A means to generate appropriate guidance on visa applications and renewals and to assist in preparing the necessary documents for application

[0757] The server provides appropriate guidance for visa application and renewal based on the latest visa and immigration rules and assists in preparing the necessary documents. The software used includes a regulatory database and a guidance generation engine. For example, the server uses the regulatory database to obtain the latest rules and provides appropriate guidance using the guidance generation engine.

[0758] Examples:

[0759] If a user types in "What is the procedure for applying for a visa?", the server will provide detailed guidance based on the latest rules. For example, "What is the procedure for applying for a visa?"

[0760] 8. Means of tailoring responses and services to users based on sentiment analysis

[0761] The server analyzes the user's emotions and provides appropriate responses and services based on the results. The software used includes an emotion analysis API, for example, to understand the user's emotions and respond accordingly.

[0762] Examples:

[0763] If a user types "It's difficult to apply for a visa," the server can analyze the sentiment and provide specific procedural guidance. For example, "Please analyze the user's sentiment from the following text and generate an appropriate response."

[0764] This system enables users to quickly understand materials and meeting content in different languages ​​and receive appropriate guidance and services.

[0765] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0766] Step 1:

[0767] Users upload pre-distributed materials to the system. The input is in the form of a PDF file or a Word document. The output is the material data stored on the server. The specific operation is that the user selects a file through the system interface and clicks the upload button.

[0768] Step 2:

[0769] The server receives the uploaded materials and uses natural language processing technology to analyze the contents of the materials. The input is the uploaded material data, and the output is the key points of the analyzed materials. Specifically, the server calls the natural language processing API to analyze the contents of the materials.

[0770] Step 3:

[0771] The server generates a summary of the material based on the analysis results. The input is the key points of the analyzed material, and the output is the generated summary. Specifically, the server uses the generative AI model to generate the summary.

[0772] Step 4:

[0773] The server translates the generated summary into the user's native language. The input is the generated summary, and the output is the translated summary. In concrete terms, the server calls a translation API to translate the summary into the user's native language.

[0774] Step 5:

[0775] The terminal captures speech during a conference through a microphone. The input is voice data, and the output is the captured voice data. In a specific operation, the terminal uses the microphone to record the voice.

[0776] Step 6:

[0777] The server converts the acquired voice data into text using voice recognition technology. The input is the acquired voice data, and the output is the text of the spoken content. In concrete terms, the server calls the voice recognition API and converts the voice data into text.

[0778] Step 7:

[0779] The server analyzes the textualized utterance. The input is the textualized utterance, and the output is the key points of the analyzed utterance. Specifically, the server analyzes the utterance using a natural language processing API.

[0780] Step 8:

[0781] The server generates a summary of the utterance based on the analysis results. The input is the key points of the analyzed utterance, and the output is the generated summary. Specifically, the server generates the summary using the generative AI model.

[0782] Step 9:

[0783] The server receives the user's input data and analyzes the emotions using emotion analysis technology. The input is the user's text or voice data, and the output is the analyzed emotional state. Specifically, the server calls the emotion analysis API and analyzes the user's emotions.

[0784] Step 10:

[0785] The server adjusts responses and services to the user based on the analysis results. The input is the analyzed emotional state, and the output is the adjusted response or service. The specific operation is for the server to perform processing to provide appropriate guidance and information.

[0786] Step 11:

[0787] The server provides appropriate guidance for visa application and renewal based on the latest visa and immigration rules, and assists in preparing the necessary documents. The input is the user's questions and information on the visa application, and the output is detailed guidance and a list of documents. In concrete terms, the server consults the regulatory database and provides appropriate guidance using a guidance generation engine.

[0788] (Application example 2)

[0789] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0790] In today's business environment, making meetings and conferences more efficient is an important issue. In particular, in meetings and conferences held inside self-driving vehicles, there is a need to efficiently share information and make decisions while on the move. However, there is a lack of means to summarize the contents of meetings in real time and provide them to participants. It is also difficult to analyze participants' emotions and provide appropriate responses and services. To solve these issues, it is necessary to analyze utterances in real time, generate summaries, and adjust responses based on emotion analysis.

[0791] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0792] In this invention, the server includes a means for summarizing emails, a means for converting keywords in the native language into Japanese sentences, a means for summarizing advance materials into the native language, a means for instantly summarizing participants' speech, a means for instantly researching difficult words, a means for informing participants of what day it is and what customs exist, a means for generating appropriate guidance for visa applications and renewals in accordance with the latest visa and immigration rules, etc., and for supporting the preparation of documents required for applications, a means for analyzing speeches of meetings and conferences held in the vehicle in real time, generating summaries and displaying them on an in-vehicle display or smart glasses, and a means for adjusting responses and services to stress and difficult situations while driving based on the analyzed emotions. This enables meetings and conferences in self-driving vehicles to be made more efficient, reduces stress for participants, and provides appropriate responses and services.

[0793] The "email summarization means" is a means for analyzing the body of an e-mail, extracting important information, and generating a summary.

[0794] The "means for converting keywords in a user's native language into Japanese sentences" refers to a means for converting keywords entered in a user's native language into Japanese sentences.

[0795] The "means for summarizing advance materials into the user's native language" refers to a means for analyzing materials distributed in advance and generating a summary of the materials in the user's native language.

[0796] The "means for instantly summarizing participant speech" refers to a means for analyzing speech during a conference in real time and generating a summary thereof.

[0797] An "instant lookup means for difficult words" is a means for instantly looking up difficult words or technical terms that come up during a meeting and providing their meanings.

[0798] A "means of telling what day it is and what customs it has" is a means of providing information about events and customs related to the current date.

[0799] "Means of generating appropriate guidance on visa applications and renewals and supporting the preparation of documents required for applications" means means of providing guidance on visa applications and renewals and supporting the preparation of necessary documents based on the latest visa and immigration rules.

[0800] "A means for analyzing in-vehicle conference or meeting speech in real time, generating a summary, and displaying it on an in-vehicle display or smart glasses" refers to a means for analyzing in-vehicle conference or meeting speech in real time, generating a summary, and displaying it on an in-vehicle display or smart glasses.

[0801] "Means for adjusting responses and services to stressful or difficult situations while driving based on analyzed emotions" refers to means for analyzing emotions from speech and behavior during a meeting and providing appropriate responses and services according to stressful or difficult situations while driving.

[0802] A system for implementing the present invention uses the following hardware and software.

[0803] Hardware used:

[0804] In-car display

[0805] Smart Glasses

[0806] microphone

[0807] High-performance processor (e.g. NVIDIA Jetson)

[0808] Software used:

[0809] Voice recognition software (e.g. Google Cloud Speech-to-Text)

[0810] Natural language processing libraries (e.g. spaCy, NLTK)

[0811] Sentiment analysis models (e.g. BERT, GPT-3)

[0812] Summary generation models (e.g. T5, BART)

[0813] Overview of what the system does:

[0814] The server first collects speech from meetings and conferences using the in-car microphone. The collected voice data is converted into text data using Google Cloud Speech-to-Text. This text data is then analyzed using a natural language processing library (spaCy or NLTK) to extract important keywords and phrases. A summary is generated based on the extracted keywords and phrases using a summary generation model (T5 or BART). This summary is then displayed on the in-car display or smart glasses.

[0815] In addition, the server inputs the generated text data into a sentiment analysis model (BERT or GPT-3) to analyze the user's emotions. Based on the analysis results, the server adjusts responses and services to stressful or difficult situations while driving. For example, if the user is feeling stressed, the server can play relaxing music.

[0816] Examples:

[0817] If someone says during a meeting, "I'd like to check the progress of the next project," the server will generate a "summary of the project's progress" and display it on the in-car display or smart glasses.

[0818] If the user says "this procedure is difficult," the server will provide "detailed guidance on the procedure."

[0819] Example prompt for a generative AI model:

[0820] Please summarize what was said at the meeting.

[0821] "Analyze the sentiment from user utterances and generate appropriate responses."

[0822] In this way, meetings and consultations inside self-driving vehicles can be made more efficient and stress-free for users.

[0823] The flow of the identification process in the application example 2 will be described with reference to FIG.

[0824] Step 1:

[0825] The server collects speech from meetings and conferences using a microphone in the car. The input is voice data, and the output is the collected voice data. This voice data is used in subsequent processing steps.

[0826] Step 2:

[0827] The server converts the collected voice data into text data using Google Cloud Speech-to-Text. The input is voice data, and the output is text data. This conversion allows the voice information to be treated as text information.

[0828] Step 3:

[0829] The server analyzes the generated text data using natural language processing libraries (spaCy or NLTK) to extract important keywords and phrases. The input is the text data, and the output is the extracted keywords and phrases. This analysis makes the main points of the meeting clear.

[0830] Step 4:

[0831] The server generates a summary based on the extracted keywords and phrases using a summary generation model (T5 or BART). The input is the keywords and phrases, and the output is the summary text. This summary is a concise summary of the contents of the meeting.

[0832] Step 5:

[0833] The server displays the generated summary text on an in-car display or smart glasses. The input is the summary text, and the output is the displayed summary information, allowing users to visually confirm the main points of the meeting.

[0834] Step 6:

[0835] The server inputs the generated text data into a sentiment analysis model (BERT or GPT-3) to analyze the user's sentiment. The input is text data, and the output is the sentiment analysis result. This analysis allows the user's emotional state to be understood.

[0836] Step 7:

[0837] The server tailors responses and services to stressful or difficult situations while driving based on the results of emotion analysis. The input is the emotion analysis result, and the output is the tailored response or service. For example, if the user is feeling stressed, the server plays relaxing music.

[0838] In this way, through a series of processes, the server can make meetings and discussions inside self-driving vehicles more efficient and reduce stress for users.

[0839] Example 3

[0840] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0841] In today's information society, users need to efficiently process and understand a lot of information. However, there is a lot of information that users cannot immediately understand, such as technical terms, abbreviations, holidays, anniversaries, and information on visa applications. There is also a demand for information that takes into account the user's emotional state and for email summaries, but there is no system that provides these in a unified manner. This means that users spend a lot of time and effort searching for and understanding information, making efficient information processing difficult.

[0842] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0843] In this invention, the server includes a means for summarizing emails, a means for converting keywords in the user's native language into Japanese sentences, a means for summarizing advance materials into the user's native language, a means for instantly summarizing speeches of participants, a means for instantly researching difficult words, a means for informing the user of what day it is and what customs exist, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules and for supporting the preparation of documents required for applications, a means for analyzing the user's emotional state and adjusting the summary content of emails based on that information, a means for analyzing words that the user does not understand, researching their meanings and uses from information sources on the Internet and providing the results, and a means for obtaining information about today's date and providing information about applicable holidays and anniversaries. This enables the user to efficiently process and understand a wide range of information.

[0844] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[0845] The "means for converting native language keywords into Japanese sentences" refers to a means for analyzing native language keywords entered by a user and converting them into appropriate Japanese sentences.

[0846] The "means for summarizing advance materials into the user's native language" refers to a means for analyzing materials provided in advance and summarizing their contents in the user's native language.

[0847] The "means for instantly summarizing participant utterances" refers to a means for analyzing in real time the contents of utterances made by participants in a meeting or conversation, and generating a summary.

[0848] An "instant lookup tool for difficult words" is a tool that analyzes technical terms and abbreviations that a user does not understand, looks up their meaning and usage from online sources, and provides the results.

[0849] "A means of informing people of what day it is and what customs apply today" is a means of obtaining information about today's date and providing information about the corresponding national holidays, anniversaries, and customs and events that take place on that day.

[0850] "Means for generating appropriate guidance regarding visa applications and renewals, and supporting the preparation of documents required for applications" refers to means for analyzing information input by users based on the latest visa and immigration rules, generating a list of required documents, and providing instructions on how to prepare each document.

[0851] The "means for analyzing the emotional state of a user and adjusting the summary content of an email based on that information" refers to a means for analyzing the emotional state of a user and adjusting the summary content of an email based on the results of that analysis.

[0852] "A means for analyzing words that a user does not understand, researching their meaning and use from information sources on the Internet, and providing the results" refers to a means for analyzing words that a user inputs that the user does not understand, researching their meaning and use from information sources on the Internet, and providing the results.

[0853] "Means for obtaining information about today's date and providing information about corresponding holidays and anniversaries" refers to means for obtaining today's date and providing information about corresponding holidays and anniversaries.

[0854] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0855] The present invention is a system for supporting a user in efficiently processing and understanding a wide range of information. A specific embodiment of this system is described below.

[0856] Terminology and acronym analysis and research

[0857] The server analyzes words that the user does not understand and investigates their meaning and usage. Specifically, it uses Python to utilize natural language processing libraries (e.g., NLTK and spaCy) to analyze the input words. Based on the analysis results, it retrieves related information from Internet sources (e.g., Wikipedia API and Google Search API) and provides it to the user.

[0858] Examples:

[0859] If a user types in "What is an API?", the server will parse the phrase and use the Wikipedia API to get information about the definition and uses of "API" and display it to the user.

[0860] Example prompt:

[0861] "What is an API?"

[0862] Survey of Japanese holidays and anniversaries

[0863] The server checks what day it is and provides the user with information about customs and events that take place on that day. Specifically, it uses Python to refer to a calendar API (e.g., Google Calendar API) or a public holiday database to obtain the relevant information.

[0864] Examples:

[0865] If a user asks "What day is it today?", the server uses the Google Calendar API to find out that today is Respect for the Aged Day and provides the user with information about its background and related events.

[0866] Example prompt:

[0867] "What day is it today?"

[0868] Visa Application and Renewal Guidance

[0869] The server generates a list of documents required for the visa application based on the information input by the user and provides instructions on how to prepare each document. Specifically, it uses Python to analyze the information input by the user and generates a list of documents required based on the pre-prepared visa application guidelines.

[0870] Examples:

[0871] If a user enters, "I would like to apply for a tourist visa," the server analyzes that information, generates a list of documents required for the tourist visa (e.g., passport copies, application form, photographs, etc.), and provides instructions on how to prepare each document and where to submit it.

[0872] Example prompt:

[0873] "What do I need to apply for a tourist visa?"

[0874] Integration of emotion engine and email summarization function

[0875] The server uses an emotion engine to analyze the user's emotional state and adjusts the email summary content based on that information. Specifically, it uses Python to leverage emotion analysis libraries (e.g. TextBlob and VADER) to analyze the user's emotional state. Based on the results, it adjusts the email summary algorithm and highlights important information.

[0876] Examples:

[0877] If the user indicates anger, the server analyzes the emotional state and adjusts the email digest function to highlight anger-related information.

[0878] Example prompt:

[0879] Please give me a summary of this email.

[0880] The above is a specific embodiment of this system. This enables the user to efficiently process and understand a wide range of information. The flow of the specific process in the third embodiment will be described with reference to FIG.

[0881] Terminology and acronym analysis and research

[0882] Step 1:

[0883] The user types words they do not understand.

[0884] Input: "What is an API?"

[0885] Output: The input word "API"

[0886] Step 2:

[0887] The server analyzes the entered words.

[0888] Input: The entered word "API"

[0889] Data processing: Use Python to analyze words using natural language processing libraries (e.g. NLTK and spaCy).

[0890] Output: Analysis results (e.g. "API stands for Application Programming Interface")

[0891] Step 3:

[0892] The server retrieves relevant information from sources on the Internet.

[0893] Input: Analysis result (e.g. "API stands for Application Programming Interface")

[0894] Data computation: Uses the Wikipedia API to obtain information about the definition and uses of "API."

[0895] Output: The information obtained (e.g. "An API provides an interface between software programs")

[0896] Step 4:

[0897] The server provides the obtained information to the user.

[0898] Input: The information obtained (e.g., "An API provides an interface between software programs.")

[0899] Output: Information displayed to the user (e.g., "An API provides an interface between software programs")

[0900] Survey of Japanese holidays and anniversaries

[0901] Step 1:

[0902] A user enters a question about today's date.

[0903] Input: "What day is it today?"

[0904] Output: Given the question "What day is it today?"

[0905] Step 2:

[0906] The server gets today's date.

[0907] Input: Current date

[0908] Data processing: Get the current date using Python.

[0909] Output: Retrieved date (e.g. September 21st)

[0910] Step 3:

[0911] The server consults the holiday database.

[0912] Input: Acquired date (e.g. September 21st)

[0913] Data calculation: Refer to the Google Calendar API or the holiday database to check the applicable holidays and anniversaries.

[0914] Output: Information about holidays and anniversaries (e.g. "Today is Respect for the Aged Day")

[0915] Step 4:

[0916] The server provides the user with information about applicable holidays and anniversaries.

[0917] Input: Holiday or anniversary information (e.g. "Today is Respect for the Aged Day")

[0918] Output: Information displayed to the user (e.g. "Today is Respect for the Aged Day. Respect for the Aged Day is a day to honor the elderly and is typically celebrated with family.")

[0919] Visa Application and Renewal Guidance

[0920] Step 1:

[0921] The user enters information about their visa application.

[0922] Input: "I want to apply for a tourist visa"

[0923] Output: Input information: "I would like to apply for a tourist visa."

[0924] Step 2:

[0925] The server parses the input information.

[0926] Input:Information entered: "I would like to apply for a tourist visa"

[0927] Data Processing: Use Python to analyze user input and identify documents required for a tourist visa.

[0928] Output: List of required documents (e.g. passport copies, application forms, photos)

[0929] Step 3:

[0930] The server generates a list of required documents.

[0931] Input: List of required documents (e.g. passport copies, application forms, photos)

[0932] Data calculation: Generate document list based on pre-prepared visa application guidelines.

[0933] Output: Generated document list

[0934] Step 4:

[0935] The server instructs the user on how to create each document.

[0936] Input: Generated document list

[0937] Output: Instructions to be displayed to the user (e.g. "Please copy all pages of your passport. Application forms can be downloaded from the embassy's website.")

[0938] Integration of emotion engine and email summarization function

[0939] Step 1:

[0940] A user requests a summary of their email.

[0941] Input: "What is the summary of this email?"

[0942] Output: Input request: "Please tell me the summary of this email."

[0943] Step 2:

[0944] The server analyzes the user's emotional state.

[0945] Input: The user's emotional state

[0946] Data processing: Analyze the emotional state using a sentiment analysis library (e.g. TextBlob or VADER).

[0947] Output: Analysis results (e.g., emotion of anger)

[0948] Step 3:

[0949] The server parses the email content.

[0950] Input: Email content

[0951] Data Calculation: Analyze the body of the email and extract important information.

[0952] Output: Extracted important information

[0953] Step 4:

[0954] The server generates a summary taking into account the emotional state.

[0955] Input: Analysis results (e.g., emotion of anger), extracted important information

[0956] Data Computation: Adjust the summarization algorithm to take into account the emotional state and generate summaries.

[0957] Output: Generated summary (e.g. "This email is about the progress of the project, highlighting any delays that may have occurred")

[0958] The above are the specific processing steps of the program of this system.

[0959] (Application example 3)

[0960] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0961] Modern brick-and-mortar stores are required to improve customer service, but there is a lack of systems that can instantly answer customer questions or provide information on events and promotions related to a particular day. There are also limited means to provide the latest visa information and application guidance for foreign tourists. Furthermore, there is no system that can analyze the emotional state of customers and provide appropriate information. To solve these issues, a system that integrates multiple functions is needed.

[0962] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0963] In this invention, the server includes a means for summarizing emails, a means for converting keywords in the native language into Japanese sentences, a means for summarizing advance materials into the native language, a means for instantly summarizing speeches of participants, a means for instantly researching difficult words, a means for informing a user of what day it is and what customs exist, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules and the like and for supporting the preparation of documents required for applications, a means for instantly answering customer questions, a means for providing event and promotion information related to a specific day, and a means for analyzing emotional states and providing appropriate information. This makes it possible to improve customer service, respond to foreign tourists, and provide information based on the customer's emotions.

[0964] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[0965] The "means for converting keywords in a user's native language into Japanese sentences" refers to a means for converting keywords in a user's native language inputted into Japanese sentences.

[0966] "Means of summarizing advance materials into one's native language" refers to means of summarizing materials prepared in advance into one's native language.

[0967] The "means for instantly summarizing participant speech" refers to a means for instantly summarizing the contents of speech by participants in a conference or event.

[0968] An "instant lookup tool for difficult words" is a tool that enables users to instantly look up words or technical terms that they do not understand and provide their meanings and uses.

[0969] "Means of teaching what day it is and what customs exist" refers to a means of providing information about what day it is and the customs and events that take place on that day.

[0970] "A means of generating appropriate guidance regarding visa applications and renewals in accordance with the latest visa and immigration rules, etc., and supporting the preparation of documents necessary for applications" refers to a means of providing guidance necessary for visa applications and renewals based on the latest visa information and immigration rules, and supporting the preparation of necessary documents.

[0971] "Means for instantly answering customer questions" refers to a means for providing instant answers to questions asked by customers in a physical store.

[0972] The "means for providing information about events and promotions related to a specific day" refers to a means for providing customers with information about events and promotions related to a specific day.

[0973] The "means for analyzing an emotional state and providing appropriate information" is a means for analyzing an emotional state of a customer and providing appropriate information based on the result of the analysis.

[0974] The system for carrying out the present invention comprises a server and a user terminal. The server includes a means for summarizing emails, a means for converting keywords in a user's native language into Japanese sentences, a means for summarizing advance materials into the user's native language, a means for instantly summarizing speeches of participants, a means for instantly researching difficult words, a means for informing a user of what day it is and what customs exist today, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules and the like, a means for supporting the preparation of documents required for applications, a means for instantly answering customer questions, a means for providing event and promotion information related to a specific day, and a means for analyzing an emotional state and providing appropriate information.

[0975] Hardware and Software Used

[0976] Hardware: Smartphones, servers

[0977] Software: Python, API request library (requests), natural language processing library (TextBlob)

[0978] Data processing and calculation

[0979] 1. Email Summarization: The server analyzes the body of an email, extracts important information, and generates a summary, allowing the user to quickly grasp the important information.

[0980] 2. Means of converting keywords in the user's native language into Japanese sentences: The server converts the keywords entered by the user in the user's native language into Japanese sentences, making it easier for foreign users to understand information in Japanese.

[0981] 3. Means for summarizing pre-prepared materials into the user's native language: The server summarizes pre-prepared materials into the user's native language, allowing the user to quickly obtain the information he or she needs in his or her native language.

[0982] 4. Means for instantly summarizing participant speech: The server instantly summarizes what participants say in meetings, events, etc. This allows users to quickly grasp the main points of the meeting.

[0983] 5. Instant search for difficult words: The server can instantly search for words or technical terms that the user does not understand and provide their meaning and usage, allowing the user to quickly understand technical information.

[0984] 6. Means of informing users of what day it is and what customs are observed: The server provides information on what day it is, the customs and events that are observed on that day, etc. This allows users to know the special information for that day.

[0985] 7. Means for generating appropriate guidance for visa application and renewal in accordance with the latest visa and immigration rules, etc., and supporting the preparation of documents required for application: The server provides guidance required for visa application and renewal based on the latest visa information and immigration rules, and supports the preparation of necessary documents. This allows foreign users to smoothly proceed with the visa application process.

[0986] 8. Means of immediate response to customer questions: The server will provide immediate answers to questions asked by customers in physical stores, thereby improving customer service.

[0987] 9. Means for Providing Event and Promotion Information Related to a Specific Day: The server provides event and promotion information related to a specific day to the customer, so that the customer can be aware of special events and promotions for that day.

[0988] 10. Means for analyzing emotional state and providing appropriate information: The server analyzes the emotional state of the customer and provides appropriate information based on the results. This allows services to be provided according to the customer's emotions.

[0989] Examples

[0990] For example, if a user asks "What documents are required for a visa application?", the server would use the following API to retrieve the information and provide the latest visa information.

[0991] Example prompt:

[0992] When a user asks "What documents do I need to apply for a visa?", use the following API to get the information to provide the latest visa information.

[0993] The flow of the specific process in the application example 3 will be described with reference to FIG.

[0994] Step 1:

[0995] A user types a question into a terminal, for example, "What documents do I need to apply for a visa?"

[0996] Input: The user's question text

[0997] Output: The question text is sent to the server.

[0998] Step 2:

[0999] The server analyzes the received question text and understands the intent of the question. It uses a natural language processing library (TextBlob) to analyze the meaning of the text.

[1000] Input: Question text

[1001] Output: Parsed question intent

[1002] Step 3:

[1003] The server generates an appropriate API request based on the parsed intent of the question, for example, to get visa information.

[1004] Input: Parsed question intent

[1005] Output: API request

[1006] Step 4:

[1007] The server sends the generated API request to an external visa information service.

[1008] Input: API request

[1009] Output: Response from the visa information service

[1010] Step 5:

[1011] The server analyzes the response received from the visa information service and extracts the necessary information.

[1012] Input: Response from the visa information service

[1013] Output: Extracted visa information

[1014] Step 6:

[1015] The server converts the extracted visa information into a format that is easy for the user to understand, for example by generating a list of required documents and providing instructions on how to complete each one.

[1016] Input: Extracted visa information

[1017] Output: Visa information for the user

[1018] Step 7:

[1019] The server transmits the generated visa information for the user to the terminal.

[1020] Input: Visa information for the user

[1021] Output: Visa information sent to the terminal

[1022] Step 8:

[1023] The terminal displays the received visa information to the user, who can then check the list of required documents and how to prepare them.

[1024] Input: Visa information sent to the terminal

[1025] Output: Visa information displayed to the user

[1026] In this way, the server provides appropriate information in response to a question entered by the user, enabling the user to quickly obtain the information he or she needs.

[1027] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1028] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1029] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1030] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[1031] [Second embodiment]

[1032] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[1033] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[1034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1035] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[1036] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1037] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1039] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1040] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1042] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1043] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1044] "Example 1"

[1045] An embodiment of the present invention provides a system as follows. First, as a means for summarizing emails, the body of the email is analyzed, important information is extracted, and a summary is generated. For example, important keywords and phrases are extracted from the body of the email, and a summary is generated based on these. Next, as a means for converting native language keywords into Japanese sentences, native language keywords entered by the user are analyzed, and corresponding Japanese sentences are generated. For example, when keywords such as "meeting" or "materials" are entered, corresponding Japanese sentences are generated.

[1046] "Example 2"

[1047] Furthermore, as a means of summarizing advance materials into the user's native language, materials distributed in advance are analyzed and summaries are generated in the user's native language. For example, materials such as meeting agendas and minutes are analyzed and summaries are provided in the user's native language. Also, as a means of instantly summarizing participants' speech, speech during a meeting is analyzed in real time and a summary is generated. For example, speech during a meeting is converted into text using speech recognition technology, and that text is analyzed to generate a summary.

[1048] "Example 3"

[1049] Furthermore, as a means of instantly researching difficult words, it analyzes words that the user does not understand and looks up their meaning and usage. For example, it looks up technical terms and abbreviations from sources on the Internet and provides the results to the user. Also, as a means of teaching users what day it is and what customs are associated with it, it looks up Japanese holidays, anniversaries, seasonal events, etc. and provides that information to the user. For example, it looks up what day it is, the customs and events that take place on that day, etc. and provides that information to the user.

[1050] "Example 4"

[1051] Finally, the system generates appropriate guidance for visa applications and renewals in line with the latest visa and immigration rules, and as a means of supporting the preparation of documents required for application, generates a list of required documents based on information input by the user and provides instructions on how to prepare each document. For example, the system generates a list of required documents based on information such as the user's visa type, period of stay, and purpose of application, and provides instructions on how to prepare each document and where to submit it.

[1052] The process flow of each embodiment will be described below.

[1053] "Example 1"

[1054] Step 1: A user email is sent to the system.

[1055] Step 2: The system analyzes the body of the email and extracts key information.

[1056] Step 3: Based on the extracted information, the system generates a summary of the email.

[1057] Step 4: The generated summary is sent to the user, who reviews the summary.

[1058] "Example 2"

[1059] Step 1: The user enters keywords in their native language into the system.

[1060] Step 2: The system analyzes the entered keywords.

[1061] Step 3: The system generates Japanese sentences that correspond to the analyzed keywords.

[1062] Step 4: The generated Japanese text is sent to the user, who confirms the text. "Example 3"

[1063] Step 1: Pre-distributed materials are sent to the system.

[1064] Step 2: The system analyzes the material and extracts key information.

[1065] Step 3: Based on the extracted information, the system generates a summary of the material in the native language.

[1066] Step 4: The generated summary is sent to the user, who reviews the summary.

[1067] "Example 4"

[1068] Step 1: User's visa application information is entered into the system.

[1069] Step 2: The system will generate a list of required documents based on the information entered.

[1070] Step 3: Provide the user with the generated list and instructions on how to create each document.

[1071] Step 4: The user follows the instructions to prepare the documents and submit the visa application.

[1072] Example 1

[1073] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1074] In today's business environment, there is a need to quickly understand the contents of emails and efficiently extract important information. Also, to facilitate communication between different languages, technology is needed to convert keywords in one's native language into other languages. Furthermore, procedures for visa applications and renewals are complicated, and appropriate guidance and support for document preparation are required. A comprehensive system to solve these problems is needed.

[1075] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1076] In this invention, the server includes a means for summarizing the email, a means for converting native language keywords into Japanese sentences, a means for analyzing the email body using a generative AI model, extracting important information, and generating a summary, and a means for converting native language keywords into Japanese sentences using the generative AI model. This allows the user to quickly understand the content of the email and efficiently extract important information. It also allows smooth communication between different languages, and appropriately supports visa application and renewal procedures.

[1077] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[1078] The "means for converting keywords in one's native language into sentences in Japanese" refers to a means for analyzing keywords in one's native language input by a user and generating corresponding sentences in Japanese.

[1079] The "means for summarizing advance materials into the native language" refers to a means for analyzing materials provided in advance and generating a summary in the native language.

[1080] The "means for instantly summarizing participant speech" refers to a means for analyzing the speech of participants during a meeting or discussion in real time and generating a summary.

[1081] An "instant search tool for difficult words" is a means for instantly searching difficult words that appear in conversations or texts, and providing their meanings and related information.

[1082] A "means of teaching what day and customs it is today" is a means of providing information about events and customs associated with a particular date.

[1083] The "Visa Application and Renewal Support Tool" is a tool that generates guidance necessary for visa applications and renewals based on the latest visa and immigration rules, and supports the preparation of documents necessary for applications.

[1084] A "generative AI model" is a model that uses artificial intelligence technology to generate and analyze text.

[1085] A "prompt sentence" is an instruction sentence input to a generative AI model, which is an input sentence to obtain a specific output.

[1086] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1087] The present invention is a system including a means for summarizing emails and a means for converting keywords in a native language into Japanese sentences. A specific embodiment of this system will be described below.

[1088] Email summary method

[1089] The server retrieves emails received by the user. It uses a mail server (e.g., Gmail API or IMAP protocol) to retrieve new emails from the user's mailbox. It uses a natural language processing library such as Python's NLTK or spaCy to analyze the body of the retrieved email. This allows it to extract important keywords and phrases from the email body. It uses a generative AI model (e.g., OpenAI's GPT-3) to generate a summary based on the extracted keywords and phrases. The generated summary is displayed in the user's email client.

[1090] Examples:

[1091] If the text of an email received by a user is "Tomorrow's meeting will discuss the progress of the new project. Please prepare materials.", the server retrieves this email and uses NLTK to extract keywords such as "meeting," "new project," "progress," and "materials." It then uses GPT-3 to generate a summary sentence that reads "Tomorrow's meeting will discuss the progress of the new project. Please prepare materials." and displays it in the user's email client.

[1092] Example prompt:

[1093] "Please summarize the body of the email below: In tomorrow's meeting, we will discuss the progress of the new project. Please prepare the materials."

[1094] A means of converting keywords in your native language into Japanese sentences

[1095] The device analyzes the native language keywords entered by the user and generates corresponding Japanese sentences. When the user enters native language keywords into the device, the device analyzes these keywords and creates a prompt sentence to input to the generative AI model. The prompt sentence is input into the generative AI model (e.g., OpenAI's GPT-3) to generate Japanese sentences. The generated Japanese sentences are displayed on the user's screen.

[1096] Examples:

[1097] If a user enters the keywords "meeting" and "documents" into the device, the device analyzes these keywords and creates a prompt sentence: "Generate a Japanese sentence containing the following keywords: meeting, documents." It then uses GPT-3 to generate the Japanese sentence "Please prepare documents for the meeting" and displays it on the user's screen.

[1098] Example prompt:

[1099] "Generate Japanese sentences containing the following keywords: meeting, materials"

[1100] In this way, the server and the terminal cooperate to generate summaries of the emails received by the user and convert keywords in the user's native language into Japanese sentences. This system allows the user to quickly understand the contents of the email and facilitates communication between different languages.

[1101] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1102] Email summary method

[1103] Step 1: Receiving email

[1104] The server obtains the email received by the user.

[1105] Input: New mail from the user's mailbox

[1106] Output: Body of retrieved email

[1107] What happens: The server uses a mail server (e.g. Gmail API or IMAP protocol) to retrieve new emails from the user's mailbox.

[1108] Step 2: Analyzing the email body

[1109] The server analyzes the body of the retrieved email.

[1110] Input: Body of the retrieved email

[1111] Output: Important keywords and phrases

[1112] How it works: The server uses natural language processing libraries such as Python's NLTK or spaCy to extract important keywords and phrases from the body of the email.

[1113] Step 3: Generate a summary

[1114] The server generates a summary based on the extracted keywords and phrases.

[1115] Input: Important keywords or phrases

[1116] Output: Generated summary

[1117] How it works: The server uses a generative AI model (e.g., OpenAI's GPT-3) to generate a summary based on the extracted keywords and phrases.

[1118] Step 4: Provide a summary

[1119] The server provides the generated summary to the user.

[1120] Input: Generated summary

[1121] Output: Summary text that will be displayed in the user's mail client

[1122] Specific operation: The server displays the generated summary in the user's email client.

[1123] A means of converting keywords in your native language into Japanese sentences

[1124] Step 1: Enter your keywords

[1125] The user inputs keywords in his / her native language into the terminal.

[1126] Input: Keywords in your native language (e.g. "meeting", "documents")

[1127] Output: The entered keyword

[1128] Specific operation: The user inputs a keyword in his / her native language into the input field of the terminal.

[1129] Step 2: Analyze keywords

[1130] The terminal analyzes the input keyword.

[1131] Input: The entered keyword

[1132] Output: A prompt to be fed to the generative AI model

[1133] Specific operation: The terminal analyzes the entered keywords and creates a prompt sentence saying, "Please generate Japanese sentences containing the following keywords: meeting, materials."

[1134] Step 3: Japanese sentence generation

[1135] The device inputs a prompt sentence into the generative AI model and generates a Japanese sentence.

[1136] Input: Prompt statement

[1137] Output: Generated Japanese sentences

[1138] Specific operation: The device inputs a prompt sentence into a generative AI model (e.g., OpenAI's GPT-3), which generates the Japanese sentence, "Please prepare materials for the meeting."

[1139] Step 4: Provide Japanese text

[1140] The terminal provides the generated Japanese text to the user.

[1141] Input: Generated Japanese sentences

[1142] Output: Japanese text displayed to the user on the screen

[1143] Specific operation: The terminal displays the generated Japanese text on the user's screen.

[1144] (Application example 1)

[1145] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1146] When drivers and passengers check their emails in autonomous vehicles, it is difficult to obtain information efficiently while ensuring driving safety. In addition, there are cases where appropriate translations are not made when converting keywords in one's native language into Japanese sentences. Furthermore, there is a lack of guidance on visa applications and renewals, making it difficult to prepare the documents required for application.

[1147] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1148] In this invention, the server includes a means for summarizing emails, a means for converting keywords in the native language into Japanese sentences, and a means for providing summaries in real time for efficiently checking emails in an autonomous vehicle. This makes it possible to efficiently check emails in an autonomous vehicle, and to quickly obtain important information while ensuring driving safety. In addition, by appropriately converting keywords in the native language into Japanese, user convenience is improved. Furthermore, by providing guidance on visa applications and renewals, it becomes easier to prepare documents required for applications.

[1149] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[1150] The "means for converting keywords in one's native language into sentences in Japanese" refers to a means for analyzing keywords in one's native language input by a user and generating corresponding sentences in Japanese.

[1151] The "means for summarizing advance materials into the native language" refers to a means for analyzing materials provided in advance and generating a summary in the native language.

[1152] The "means for instantly summarizing participant speech" refers to a means for summarizing in real time what is said by participants in a meeting or event.

[1153] An "instant lookup tool for difficult words" is a tool that instantly looks up difficult words and technical terms and provides their meanings and explanations.

[1154] A "means of teaching what day and customs it is today" is a means of providing information about events and customs associated with a particular date.

[1155] "Means of generating appropriate guidance on visa applications and renewals and supporting the preparation of documents required for applications" means means of providing guidance on visa applications and renewals and supporting the preparation of necessary documents based on the latest visa and immigration rules.

[1156] A "means for providing a real-time summary for efficiently checking emails within an autonomous vehicle" is a means for providing a real-time summary of emails when a driver or passenger checks emails within an autonomous vehicle.

[1157] The present invention relates to a system for enabling drivers and passengers in autonomous vehicles to efficiently check emails and convert keywords in their native language into Japanese sentences. A specific embodiment for implementing this system is described below.

[1158] System Configuration

[1159] The system consists of the following main components:

[1160] 1. Email summary methods:

[1161] The server parses the body of the email, extracts key information and generates a summary, using the natural language processing library spaCy and the Hugging Face Transformer model.

[1162] 2. How to convert keywords in your native language into Japanese sentences:

[1163] The server analyzes the native language keywords entered by the user and generates corresponding Japanese sentences using a predefined dictionary.

[1164] 3. A way to provide real-time summaries for efficient email review in autonomous vehicles:

[1165] The server provides real-time summaries of emails, allowing drivers and passengers to safely check their emails using their smartphones or head-mounted displays.

[1166] Program processing explanation

[1167] To summarize the email, the server first parses the body of the email using the natural language processing library spaCy to extract the email's key information, then uses the Hugging Face Transformer model to generate a summary based on the extracted information.

[1168] To convert native language keywords into Japanese sentences, the server analyzes the keywords entered by the user and generates the corresponding Japanese sentences using a predefined dictionary.

[1169] The server then sends the generated summary to a smartphone or head-mounted display as a means of providing a real-time summary for efficient email review within the autonomous vehicle, allowing the driver or passengers to safely review their email.

[1170] Examples

[1171] For example, suppose a user receives the following email while in an autonomous vehicle:

[1172] Email body:

[1173] "We have a meeting scheduled for tomorrow. Please bring the necessary documents."

[1174] The summary of this email is as follows:

[1175] summary:

[1176] "We have a meeting scheduled for tomorrow. Please bring the necessary documents."

[1177] Also, if a user enters the keywords "meeting" and "document," the system will convert this into Japanese.

[1178] Translated keywords:

[1179] "Meetings, materials"

[1180] Example prompt for a generative AI model:

[1181] Email body: "We have a meeting scheduled for tomorrow. Please bring the necessary documents."

[1182] Keywords: ["meeting", "document"]

[1183] In this way, the system will streamline email checking in autonomous vehicles, enabling users to quickly obtain important information while ensuring driving safety. In addition, by appropriately converting keywords in the user's native language into Japanese, user convenience will be improved.

[1184] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1185] Step 1:

[1186] The server receives the email body from the user. As input, the email body is provided. The server parses the email body using spaCy, a natural language processing library, and extracts key information. As output, the extracted key information is provided.

[1187] Step 2:

[1188] The server generates a summary based on the extracted key information using the Hugging Face Transformer model. As input, the key information extracted in step 1 is provided. The server inputs the prompt sentence into the summary generation model and generates a summary. As output, the generated summary is obtained.

[1189] Step 3:

[1190] A user inputs keywords in his / her native language. As input, keywords in the native language are provided. The server converts these keywords into Japanese sentences using a predefined dictionary. As output, the converted Japanese sentences are obtained.

[1191] Step 4:

[1192] The server sends the generated summary and the converted Japanese text to a smartphone or head-mounted display. As input, the summary generated in step 2 and the Japanese text converted in step 3 are provided. The server sends this information to the terminal so that the user can confirm it. As output, the summary and Japanese text are displayed on the user's terminal.

[1193] Step 5:

[1194] The user checks the summary and the Japanese text on the terminal. The summary and the Japanese text sent from the server are provided as input. The user checks this information and takes the next action if necessary. The output is the user's understanding and the next action.

[1195] Example 2

[1196] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[1197] In today's business environment, it is important to quickly and accurately grasp the contents of meetings and documents. However, it is difficult to understand and summarize documents and meeting speech provided in different languages ​​in real time. It is also complex to provide appropriate information on visa applications and renewals and support the preparation of necessary documents. To solve these problems, a system that combines advanced natural language processing technology and speech recognition technology is required.

[1198] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for summarizing emails, a means for converting keywords in the native language into Japanese sentences, a means for summarizing advance materials into the native language, a means for instantly summarizing the speech of participants, a means for instantly researching difficult words, a means for informing what day it is today and what customs are there, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules, etc., and supporting the preparation of documents required for applications, a means for analyzing materials distributed in advance and generating summaries in the native language, a means for converting speech during a meeting into text using speech recognition technology and analyzing the text to generate summaries, and a means for analyzing materials and speech using a generation AI model and providing summaries in the user's native language. This makes it possible to quickly and accurately understand and summarize materials provided in different languages ​​and speech at meetings. In addition, it is possible to appropriately provide information on visa applications and renewals and support the preparation of necessary documents.

[1199] The "email summarization means" is a means for analyzing the body of an e-mail, extracting important information, and generating a concise summary.

[1200] The "means for converting keywords in the user's native language into Japanese sentences" refers to a means for converting keywords input in the user's native language into Japanese sentences.

[1201] The "means for summarizing advance materials in the user's native language" is a means for analyzing materials distributed in advance and generating a summary of the materials in the user's native language.

[1202] The "means for instantly summarizing participant speech" refers to a means for analyzing speech during a conference in real time and generating a summary thereof.

[1203] The "instant search means for difficult words" is a means for instantly searching difficult words that appear in meetings or documents, and providing their meanings and related information.

[1204] A "means of teaching what day and customs it is today" is a means of providing information about events and customs associated with a particular date.

[1205] The "Visa Application and Renewal Support Tool" is a tool that generates appropriate guidance based on the latest visa and immigration rules and supports the preparation of documents required for visa applications and renewals.

[1206] A "generative AI model" is an artificial intelligence model used for natural language processing and data analysis, such as for summarizing and translating text.

[1207] A "prompt" is an instruction entered into a generative AI model to cause it to perform a specific task.

[1208] "Voice recognition technology" is a technology that converts voice data into text data.

[1209] "Natural language processing technology" is technology for understanding, analyzing, and generating human language.

[1210] This invention is a system for quickly and accurately understanding and summarizing materials and conference speeches provided in different languages, and also includes a function for providing appropriate information on visa applications and renewals and supporting the preparation of necessary documents.

[1211] System Configuration

[1212] The system is composed of three main components: a server, a terminal, and a user. The server uses a generative AI model to analyze materials and speech and provides summaries in the user's native language. The terminal converts speech during meetings into text using speech recognition technology and sends the text to the server. Users can upload materials and summarize meeting speech in real time through the system.

[1213] Hardware and Software Used

[1214] Server: A server with high-performance computing power is required, preferably equipped with a GPU to run the generative AI model.

[1215] Terminal: A terminal with a microphone to execute the voice recognition technology and communication capabilities to send the converted text data to the server is required.

[1216] software:

[1217] Generative AI models: Use natural language processing models such as OpenAI's GPT-4 or Google's BERT.

[1218] Speech Recognition Technology: Using voice recognition software such as Google Speech-to-Text and IBM Watson Speech to Text.

[1219] Translation API: Use a translation API such as DeepL or Google Translate.

[1220] Program Processing

[1221] The server receives the advance materials uploaded by the user, analyzes the contents of the materials using natural language processing technology, and generates a summary. The generated summary is translated into the user's native language using a translation API and provided to the user.

[1222] The device converts speech during the meeting into text in real time using speech recognition technology and sends the text to the server. The server analyzes the received text and generates a summary. The generated summary is translated into the user's native language using a translation API and provided to the user in real time.

[1223] Examples

[1224] For example, when a user uploads English minutes, the server receives them, analyzes them, generates a summary in Japanese, and provides it to the user. In addition, the device recognizes speech spoken in English during a meeting, converts it into text, and sends it to the server. The server analyzes the text, generates a summary in Japanese, and provides it to the user in real time.

[1225] Example of prompt when summarizing minutes:

[1226] "Please summarize the following English minutes in Japanese."

[1227] Example prompts for summarizing meeting utterances:

[1228] "Please summarize the following English statements in Japanese."

[1229] The system will enable users to quickly and accurately understand and summarize materials and conference speeches provided in different languages, and will also provide appropriate information on visa applications and renewals, and assist with the preparation of necessary documents.

[1230] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1231] Step 1:

[1232] The user uploads advance materials. The user uploads materials such as meeting agendas and minutes through the system interface. The input is a document file such as PDF or Word, and the output is the material data sent to the server.

[1233] Step 2:

[1234] The server receives the materials. The server stores the uploaded materials and prepares them for analysis. The input is the material data sent by the user, and the output is the material data to be analyzed.

[1235] Step 3:

[1236] The server analyzes the document. The server uses natural language processing technology to understand the content of the document. In this case, a generative AI model (for example, GPT-4 or BERT) is used. The input is the document data to be analyzed, and the output is a summary of the document.

[1237] Step 4:

[1238] The server generates the summary. It uses a generative AI model to extract the main points of the material and create a concise summary. The input is the analyzed material data, and the output is the generated summary data.

[1239] Step 5:

[1240] The server translates the summary into the user's native language. Using a translation API (e.g., DeepL or Google Translate), the generated summary is translated into the user's native language. The input is the generated summary data, and the output is the translated summary data.

[1241] Step 6:

[1242] The server provides the translated summary to the user, who can view the summary through the system's interface. The input is the translated summary data, and the output is the summary information provided to the user.

[1243] Step 7:

[1244] The terminal recognizes speech during the meeting. A microphone installed in the meeting room picks up the speech and obtains the voice data. The input is the voice data during the meeting, and the output is the voice data sent to the voice recognition software.

[1245] Step 8:

[1246] The device converts the voice to text. It uses voice recognition software such as Google Speech-to-Text or IBM Watson Speech to Text to convert the voice data into text data. The input is voice data and the output is text data.

[1247] Step 9:

[1248] The terminal transmits the converted speech to the server. A communication protocol with low latency is used to send data to the server in real time. The input is text data, and the output is text data sent to the server.

[1249] Step 10:

[1250] The server analyzes the received text. It uses natural language processing technology to understand the content of the speech. The input is text data, and the output is the analyzed speech data.

[1251] Step 11:

[1252] The server generates a summary of the utterance. It uses a generative AI model to extract the main points of the utterance and create a concise summary. The input is the analyzed utterance data, and the output is the generated summary data.

[1253] Step 12:

[1254] The server translates the generated summary into the user's native language using a translation API such as DeepL or Google Translate. The input is the generated summary data, and the output is the translated summary data.

[1255] Step 13:

[1256] The server provides the translated summary to the user, who can view the summary in real time. The input is the translated summary data, and the output is the summary information provided to the user.

[1257] (Application example 2)

[1258] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1259] In today's brick-and-mortar stores, it is important for store clerks to respond quickly and accurately to customer requests and questions. However, language barriers and an overload of information can make it difficult for store clerks to accurately understand what customers are saying and respond appropriately. In addition, when store clerks are required to provide multilingual support, it is difficult for them to translate and understand in real time. This can lead to lower customer satisfaction and delayed responses.

[1260] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server includes a means for summarizing emails, a means for converting keywords in the native language into Japanese sentences, a means for summarizing advance materials into the native language, a means for instantly summarizing the speech of participants, a means for instantly researching difficult words, a means for informing what day it is today and what customs exist, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules, etc., and for supporting the preparation of documents required for application, a means for real-time speech recognition and text conversion of customer speech, a means for analyzing the text content of the speech and generating a summary, and a means for translating the summary into multiple languages ​​and displaying it. This enables store clerks to quickly and accurately understand the content of customer speech and to instantly provide multilingual support.

[1261] The "email summarization means" is a means for analyzing the body of an e-mail, extracting important information, and generating a summary.

[1262] The "means for converting keywords in a user's native language into Japanese sentences" refers to a means for converting keywords entered in a user's native language into Japanese sentences.

[1263] The "means for summarizing advance materials into the user's native language" refers to a means for analyzing materials distributed in advance and generating a summary of the materials in the user's native language.

[1264] The "means for instantly summarizing participant speech" refers to a means for analyzing speech during a conference in real time and generating a summary thereof.

[1265] An "instant lookup tool for difficult words" is a means for instantly looking up difficult words or technical terms that appear in a conversation and providing their meaning.

[1266] A "means of informing users of today's events and customs related to the current date" is a means of informing users of today's events and customs related to the current date.

[1267] "Visa Application and Renewal Support Tool" is a tool that provides guidance necessary for visa applications and renewals based on the latest visa and immigration rules, and supports the preparation of necessary documents.

[1268] "Means for performing speech recognition on a customer's speech in real time and converting the speech into text" refers to means for performing speech recognition on a customer's speech in real time and converting the content into text.

[1269] The "means for analyzing the textualized content of an utterance and generating a summary" refers to a means for analyzing the textualized content of an utterance and generating a summary thereof.

[1270] The "means for translating and displaying a summary in multiple languages" refers to a means for translating the generated summary into multiple languages ​​and displaying it to the user.

[1271] A system for carrying out this invention has the following configuration: The server includes means for summarizing emails, means for converting keywords in a person's native language into Japanese sentences, means for summarizing advance materials into the person's native language, means for instantly summarizing speeches of participants, means for instantly researching difficult words, means for informing participants what day it is and what customs exist today, means for supporting visa applications and renewals, means for speech recognition and text conversion of customer speech in real time, means for analyzing the content of the text conversion and generating a summary, and means for translating and displaying the summary into multiple languages.

[1272] Hardware and Software Used

[1273] Hardware

[1274] Smart glasses: A device with a display to capture customer speech and display a summary.

[1275] Microphone: A voice input device for capturing customer speech.

[1276] software

[1277] Speech recognition engine: Uses Google Cloud Speech-to-Text API.

[1278] Summary generation model: Uses OpenAI GPT-4.

[1279] Translation engine: Uses Google Cloud Translation API.

[1280] Data processing and calculation

[1281] 1. Voice capture: The microphone in the smart glasses captures the customer's speech, which then captures the customer's speech as voice data.

[1282] 2. Speech recognition: The captured voice data is converted to text using the Google Cloud Speech-to-Text API, which converts the voice data into text data.

[1283] 3. Summary generation: The converted speech is analyzed using OpenAI GPT-4 to generate a summary, converting long speech into a concise summary.

[1284] 4. Translation: The summarized text is translated into the clerk’s native language using the Google Cloud Translation API, making the summary multilingual.

[1285] 5. Display: The translated summary is displayed on the smart glasses display, allowing the store attendant to quickly and accurately understand the customer's request.

[1286] Examples

[1287] For example, consider a sales associate interacting with an English-speaking customer whose native language is Japanese. If the customer says, "I am looking for a red dress in size M," the system would:

[1288] 1. The microphone in the smart glasses captures the customer's speech.

[1289] 2. Convert the audio data into text using the Google Cloud Speech-to-Text API.

[1290] 3. Analyze the transcribed speech using OpenAI GPT-4 and generate a summary.

[1291] 4. Translate the summary into Japanese using the Google Cloud Translation API.

[1292] 5. Display the translated summary “Looking for a red dress in size medium” on the smart glasses display.

[1293] Examples of prompt statements

[1294] Customer utterance: "I am looking for a red dress in size M"

[1295] Generated summary: "Looking for a red dress in size medium"

[1296] In this way, the salesperson can quickly and accurately understand the customer's request and respond appropriately.

[1297] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1298] Step 1:

[1299] The microphone in the smart glasses captures the customer's speech. The input is the customer's voice data, and the output is the captured voice data. The smart glasses record the customer's speech in real time and send the voice data to the server.

[1300] Step 2:

[1301] The server converts the captured voice data into text data using the Google Cloud Speech-to-Text API. The input is voice data and the output is text data. The server calls a speech recognition engine to analyze the voice data and generate the corresponding text.

[1302] Step 3:

[1303] The server analyzes the textual content of the utterance using OpenAI GPT-4 and generates a summary. The input is the text data and the output is the summarized text. The server uses a generative AI model to extract the important parts of the text data and create a concise summary.

[1304] Step 4:

[1305] The server translates the summarized text into the clerk's native language using the Google Cloud Translation API. The input is the summarized text and the output is the translated text. The server calls the translation engine to make the summarized text multilingual.

[1306] Step 5:

[1307] The server displays the translated summary on the display of the smart glasses. The input is the translated text and the output is the summary displayed on the display of the smart glasses. The smart glasses display the translated text received from the server on their display, allowing the store clerk to quickly understand the customer's request.

[1308] Example 3

[1309] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1310] In today's information society, users need to quickly and accurately obtain a wide variety of information. However, when faced with difficult words such as technical terms, abbreviations, and foreign languages, it is difficult to immediately understand their meaning and use. In addition, in daily life and business, it is important to appropriately obtain and respond to information about holidays, anniversaries, and visa applications, but there is no system that provides this information centrally. Therefore, there is a need for a system that can instantly look up the meaning and use of words that users do not understand and provide information about holidays, anniversaries, and visa applications.

[1311] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1312] In this invention, the server includes a means for analyzing words that the user does not understand and investigating their meanings and uses, a means for providing information on Japanese holidays, anniversaries, and seasonal events, and a means for generating a list of necessary documents based on information input by the user and instructing how to prepare each document. This allows the user to instantly understand the meanings and uses of difficult words and obtain information on holidays, anniversaries, and visa applications in a centralized manner.

[1313] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[1314] The "means for converting keywords in a user's native language into Japanese sentences" refers to a means for converting keywords entered in a user's native language into Japanese sentences.

[1315] A "means for summarizing advance materials into a user's native language" is a means for summarizing advance provided materials into a user's native language.

[1316] The "means for instantly summarizing participant utterances" refers to a means for instantly summarizing the contents of utterances made by participants in a conference or dialogue.

[1317] An "instant lookup means for difficult words" is a means for analyzing words that a user does not understand and instantly looking up their meaning and usage.

[1318] "Means of teaching what day it is and what customs it has" is a means of providing information about holidays, anniversaries, and seasonal events related to the current date.

[1319] "Means to generate appropriate guidance on visa applications and renewals and support the preparation of documents required for applications" refers to means to generate a list of documents required for visa applications and renewals based on the latest visa and immigration rules, and to provide guidance on how to prepare each document.

[1320] "A means for analyzing words that a user does not understand and investigating their meaning and use" refers to a means for analyzing difficult words entered by a user and investigating their meaning and use from information sources on the Internet.

[1321] "Means for providing information about Japanese holidays, anniversaries, and seasonal events" refers to means for retrieving information about Japanese holidays, anniversaries, and seasonal events from a database and providing it to a user.

[1322] "A means for generating a list of required documents based on information input by the user and providing instructions on how to prepare each document" refers to a means for generating a list of required documents based on information input by the user, such as the type of visa, period of stay, and purpose of the application, and for providing instructions on how to prepare each document and where to submit it.

[1323] This invention is a system that analyzes words and technical terms that users do not understand, and investigates and provides their meaning and usage. It also provides information on Japanese holidays, anniversaries, and seasonal events, and generates guidance on visa applications and renewals.

[1324] System Configuration

[1325] Hardware and Software

[1326] Server: A central processing unit for data analysis and information provision. It uses programming languages ​​such as Python and Java, and MySQL as the database.

[1327] Terminal: A device on which a user enters information and displays the results. This includes PCs, smartphones, tablets, etc. HTML, CSS, and JavaScript are used for the front end.

[1328] Internet: A network for data communication between servers and terminals.

[1329] Program Processing

[1330] Word analysis and research

[1331] 1. The user types a word that the user does not understand into a terminal input field.

[1332] 2. The device sends the input words to the server, specifically by sending the data using an HTTP request.

[1333] 3. The server analyzes the received words. It uses natural language processing (NLP) techniques, such as the Python NLTK library, to analyze them. It then calls the Wikipedia API or the API of a technical dictionary to investigate the meaning of the words.

[1334] 4. The server sends the investigation results in JSON format to the terminal, including, for example, "AI is an abbreviation for Artificial Intelligence, a technology that allows machines to mimic human intelligence."

[1335] 5. The device parses the received JSON data and displays it in a user-friendly format, using HTML and CSS to display the results on the screen.

[1336] Providing information on holidays and anniversaries

[1337] 1. The user types "What day is it today?" into the input field on the terminal.

[1338] 2. The device sends the input to the server, specifically by sending the data using an HTTP request.

[1339] 3. The server looks up a database of Japanese holidays and anniversaries (for example, a MySQL database) and uses an SQL query to find the holidays and anniversaries that fall on today's date.

[1340] 4. The server sends the acquired information in JSON format to the terminal, including, for example, "Today is Respect for the Aged Day. Respect for the Aged Day is a day to respect the elderly and celebrate longevity."

[1341] 5. The device parses the received JSON data and displays it in a user-friendly format, using HTML and CSS to display the results on the screen.

[1342] Visa Application and Renewal Guidance

[1343] 1. The user enters "Tourist visa, 3 months, tourist purpose" into the input field on the terminal.

[1344] 2. The device sends the input to the server, specifically by sending the data using an HTTP request.

[1345] 3. The server generates a list of required documents based on the input information. For example, it could use Python logic to generate a list like "To apply for a tourist visa, the following documents are required: application form, copy of passport, proof of place of stay."

[1346] 4. The server generates detailed guidance on how to prepare each document and where to submit it, for example, "Please download the application form from the official website and fill in the necessary information. Please make a copy of every page of your passport."

[1347] 5. The server sends the generated guidance in JSON format to the device.

[1348] 6. The device parses the received JSON data and displays it in a user-friendly format, using HTML and CSS to display the results on the screen.

[1349] Examples of concrete examples and prompts

[1350] Examples of word analysis and research

[1351] A user types, "What is AI?"

[1352] The device sends "AI" to the server.

[1353] The server investigates the meaning of "AI" and sends the result to the terminal, stating that "AI is an abbreviation for Artificial Intelligence, a technology that allows machines to imitate human intelligence."

[1354] The terminal displays the results to the user.

[1355] Examples of providing information about holidays and anniversaries

[1356] The user types, "What day is it today?"

[1357] The terminal sends input to the server.

[1358] The server sends the result to the terminal, stating, "Today is Respect for the Aged Day. Respect for the Aged Day is a day to respect the elderly and celebrate longevity."

[1359] The terminal displays the results to the user.

[1360] Examples of guidance for visa application and renewal

[1361] The user enters "Tourist visa, 3 months, tourist purpose."

[1362] The terminal sends input to the server.

[1363] The server sends the result to the terminal as "The following documents are required to apply for a tourist visa: application form, copy of passport, and proof of place of stay."

[1364] The terminal displays the results to the user.

[1365] Examples of prompt statements

[1366] "What is AI?"

[1367] "Tell me what day it is."

[1368] Please tell me what documents I need to apply for a tourist visa.

[1369] In this way, the system searches for necessary information based on the user's input and provides it in an easy-to-understand manner. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1370] Word analysis and research

[1371] Step 1:

[1372] The user types a word that the device does not understand into an input field.

[1373] Input: What the user types (e.g. "What is AI?")

[1374] Output: Words typed into the terminal

[1375] Step 2:

[1376] The device sends the input words to the server, specifically by sending the data using an HTTP request.

[1377] Input: Words typed into the terminal

[1378] Output: The HTTP request sent to the server.

[1379] Step 3:

[1380] The server analyzes the received words using natural language processing (NLP) techniques, such as the Python NLTK library.

[1381] Input: The word sent to the server

[1382] Output: Meaning and usage of analyzed words

[1383] Step 4:

[1384] The server calls the Wikipedia API or a technical dictionary API to look up the meaning of words.

[1385] Input: Parsed words

[1386] Output: Research results (e.g. "AI stands for Artificial Intelligence, a technology that allows machines to mimic human intelligence")

[1387] Step 5:

[1388] The server sends the investigation results to the terminal in JSON format.

[1389] Input: Survey results

[1390] Output: JSON data sent to the terminal

[1391] Step 6:

[1392] The device parses the received JSON data and displays it in a user-friendly format, using HTML and CSS to display the results on the screen.

[1393] Input: JSON data sent to the terminal

[1394] Output: Findings displayed to the user

[1395] Providing information on holidays and anniversaries

[1396] Step 1:

[1397] The user types "What day is it today?" into the device's input field.

[1398] Input: The question the user types in (e.g. "What day is it today?")

[1399] Output: Question typed into the terminal

[1400] Step 2:

[1401] The device sends input to the server, specifically by sending data using an HTTP request.

[1402] Input: Question typed into the terminal

[1403] Output: The HTTP request sent to the server.

[1404] Step 3:

[1405] The server looks up a database of Japanese holidays and anniversaries (for example, a MySQL database) and uses an SQL query to find the holidays and anniversaries that fall on today's date.

[1406] Input: The question sent to the server

[1407] Output: Holiday and anniversary information retrieved from the database

[1408] Step 4:

[1409] The server sends the acquired information to the terminal in JSON format.

[1410] Input: Information retrieved from a database

[1411] Output: JSON data sent to the terminal

[1412] Step 5:

[1413] The device parses the received JSON data and displays it in a user-friendly format, using HTML and CSS to display the results on the screen.

[1414] Input: JSON data sent to the terminal

[1415] Output: Holiday and anniversary information displayed to the user

[1416] Visa Application and Renewal Guidance

[1417] Step 1:

[1418] The user types "Tourist visa, 3 months, tourist purpose" into the device's input field.

[1419] Input: Visa information entered by the user (e.g. "Tourist Visa, 3 months, tourist purpose")

[1420] Output: Visa information entered into the terminal

[1421] Step 2:

[1422] The device sends input to the server, specifically by sending data using an HTTP request.

[1423] Input: Visa information entered into the terminal

[1424] Output: The HTTP request sent to the server.

[1425] Step 3:

[1426] The server generates a list of required documents based on the input information, for example using Python logic to generate a list like "To apply for a tourist visa, the following documents are required: application form, copy of passport, proof of place of stay".

[1427] Input: Visa information sent to the server

[1428] Output: A list of generated documents

[1429] Step 4:

[1430] The server generates detailed guidance on how to prepare each document and where to submit it, for example, "Please download the application form from the official website and fill in the necessary information. Please make a copy of every page of your passport."

[1431] Input: A list of generated documents

[1432] Output: Detailed guidance

[1433] Step 5:

[1434] The server generates guidance and sends it to the terminal in JSON format.

[1435] Input:Detailed Guidance

[1436] Output: JSON data sent to the terminal

[1437] Step 6:

[1438] The device parses the received JSON data and displays it in a user-friendly format, using HTML and CSS to display the results on the screen.

[1439] Input: JSON data sent to the terminal

[1440] Output: Guidance displayed to the user

[1441] (Application example 3)

[1442] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1443] Today's consumers want to instantly obtain detailed information about products and special event information to improve their shopping experience in physical stores. However, it is difficult to immediately understand product descriptions that contain many technical terms and abbreviations, or event information held in stores. In addition, foreign tourists need to quickly and accurately obtain information about visa applications and renewals. A system that can solve these problems is needed.

[1444] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1445] In this invention, the server includes a means for summarizing emails, a means for converting keywords in the native language into Japanese sentences, a means for summarizing advance materials into the native language, a means for instantly summarizing speeches of participants, a means for instantly researching difficult words, a means for informing a participant of what day it is and what customs are in place today, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules, and for supporting the preparation of documents required for applications, a means for scanning technical terms and abbreviations contained in product packages and descriptions with a camera and instantly researching and displaying their meanings and uses, a means for providing information on today's holidays and anniversaries, special events and sales taking place in stores, and a means for providing guidance on visa applications and renewals for foreign tourists, and for instructing on a list of necessary documents and how to prepare them. This allows consumers to improve their shopping experience in physical stores, and allows foreign tourists to quickly and accurately obtain information on visa applications and renewals.

[1446] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[1447] The "means for converting keywords in a user's native language into Japanese sentences" refers to a means for converting keywords entered in a user's native language into Japanese sentences.

[1448] A "means for summarizing advance materials into a user's native language" is a means for summarizing advance provided materials into a user's native language.

[1449] The "means for instantly summarizing participant speech" refers to a means for summarizing participant speech in real time during a meeting or discussion.

[1450] An "instant lookup tool for difficult words" is a tool that allows users to instantly look up technical terms or abbreviations that they do not understand and provide their meanings and uses.

[1451] "Means of teaching what day it is and what customs exist" refers to a means of providing information about what day it is and the customs and events that take place on that day.

[1452] "A means of generating appropriate guidance regarding visa applications and renewals in accordance with the latest visa and immigration rules, etc., and supporting the preparation of documents required for applications" refers to a means of providing guidance regarding visa applications and renewals based on the latest visa and immigration rules, and supporting the preparation of necessary documents.

[1453] "A means for scanning technical terms and abbreviations contained in product packaging or descriptions with a camera, and instantly researching and displaying their meanings and uses" refers to a means for scanning technical terms and abbreviations contained in product packaging or descriptions with a camera, and instantly researching and displaying their meanings and uses to the user.

[1454] "Means for providing information about today's holidays and anniversaries, and about special events and sales taking place at stores" refers to means for providing users with information about today's holidays and anniversaries, and about special events and sales taking place at stores.

[1455] "Means of providing guidance to foreign tourists regarding visa applications and renewals, and instructing them on the list of required documents and how to prepare them" means means of providing guidance to foreign tourists regarding visa applications and renewals, and instructing them on the list of required documents and how to prepare them.

[1456] The system for implementing this invention uses the following hardware and software. The hardware uses a smartphone or smart glasses. The software uses a camera API, a natural language processing library (e.g., spaCy), a database (e.g., Firebase), and a generative AI model (e.g., GPT-4).

[1457] System Configuration

[1458] 1. Email summary methods:

[1459] The server analyzes the body of the email, extracts important information, and generates a summary, which allows the user to quickly grasp the important information.

[1460] 2. How to convert keywords in your native language into Japanese sentences:

[1461] The server converts keywords entered in the user's native language into Japanese sentences, making it easier for foreign users to understand information in Japanese.

[1462] 3. How to summarize the advance materials in your native language:

[1463] The server summarizes the pre-provided materials in the user's native language, allowing the user to quickly understand the contents of the materials.

[1464] 4. Means of immediate summary of participants' speech:

[1465] The server summarizes the speech of participants during meetings and discussions in real time, allowing users to efficiently grasp the contents of the meetings.

[1466] 5. Quick lookup for difficult words:

[1467] The server instantly searches for technical terms and abbreviations that the user does not understand and provides their meaning and usage, thereby enabling the user to easily understand technical information.

[1468] 6. Ways to teach what day it is and what customs it has:

[1469] The server provides information about what day it is, customs and events taking place on that day, etc. This allows users to obtain information that is useful in their daily lives.

[1470] 7. To generate appropriate guidance on visa application and renewal in line with the latest visa and immigration regulations, and to assist in preparing the necessary documents for application:

[1471] The server provides guidance on visa applications and renewals based on the latest visa and immigration regulations, and assists in preparing the necessary documents, allowing foreign users to smoothly apply for and renew their visas.

[1472] 8. A method for scanning technical terms and abbreviations in product packaging and descriptions with a camera and instantly researching and displaying their meaning and usage:

[1473] The device uses a camera to scan technical terms and abbreviations contained in product packaging and descriptions, and instantly researches their meanings and uses and displays them to the user, allowing the user to instantly obtain detailed information about the product.

[1474] 9. Ways to share information about current holidays, anniversaries, special events and sales happening in your store:

[1475] The server provides the user with information about today's holidays and anniversaries, as well as special events and sales taking place in stores, making it easier for the user to plan their shopping.

[1476] 10. Providing guidance to foreign tourists on visa applications and renewals, listing the necessary documents and providing instructions on how to complete them:

[1477] The server provides guidance to foreign tourists on visa applications and renewals, including a list of required documents and instructions on how to prepare them, allowing foreign tourists to efficiently apply for and renew visas.

[1478] Examples of concrete examples and prompts

[1479] Examples

[1480] Instant terminology research:

[1481] When a user uses the smart glasses to scan a product package, the term "gluten-free" appears and its meaning and uses are instantly explained.

[1482] Today's event information:

[1483] When a user opens the app, they are shown the information, "Today is Halloween. Special sales are happening at stores near you."

[1484] Visa Application Guidance:

[1485] When a foreign tourist enters "tourist visa, 3 months, tourism purpose" into the app, a list of required documents and instructions on how to prepare them are displayed.

[1486] Examples of prompt statements

[1487] Instant terminology research:

[1488] "Research the meaning and use of this term: gluten-free."

[1489] Today's event information:

[1490] "Please let me know today's holidays, anniversaries, and store events."

[1491] Visa Application Guidance:

[1492] "Please tell me the documents required to apply for a tourist visa and how to fill them out. I will be staying for 3 months and my purpose is tourism."

[1493] In this way, the system can provide users with a variety of information instantly, improving their shopping experience in brick-and-mortar stores, and providing foreign tourists with quick and accurate information on visa applications and renewals.

[1494] The flow of the specific process in the application example 3 will be described with reference to FIG.

[1495] Step 1:

[1496] A user scans a product package using a smartphone or smart glasses.

[1497] Input: Product packaging image

[1498] Output: Extracted text

[1499] Specific operation: The device's camera API is used to obtain an image of the product packaging, and text is extracted using OCR (optical character recognition) technology.

[1500] Step 2:

[1501] The server analyzes the extracted text to identify technical terms and abbreviations.

[1502] Input: Extracted text

[1503] Output: Identified terms and abbreviations

[1504] What happens: The server uses a natural language processing library (e.g. spaCy) to analyze the text and identify jargon and abbreviations.

[1505] Step 3:

[1506] The server investigates the meaning and usage of specified technical terms and abbreviations from Internet sources.

[1507] Input: Identified terminology or abbreviation

[1508] Output: Meaning and usage of technical terms and abbreviations

[1509] Specific Behavior: The server uses Internet resources (e.g., Wikipedia API) to look up the meaning and usage of identified terms and abbreviations.

[1510] Step 4:

[1511] The server displays the results of the investigation to the user.

[1512] Input: Meaning and use of technical terms and abbreviations

[1513] Output: Information displayed to the user

[1514] Specific operation: The server transmits the investigation result to the terminal, and the terminal displays the investigation result to the user.

[1515] Step 5:

[1516] The server provides information about today's holidays and anniversaries, as well as special events and sales happening in stores.

[1517] Input: Current date and user location

[1518] Output: Holidays, anniversaries, and event information

[1519] Specific operation: The server retrieves information about today's holidays, anniversaries, and store events from a database (e.g. Firebase) and provides relevant information based on the user's location.

[1520] Step 6:

[1521] The server provides guidance to foreign tourists on visa applications and renewals.

[1522] Input: User's visa type, duration of stay, purpose of application

[1523] Output: List of required documents and how to create them

[1524] How it works: The server uses a generative AI model (e.g. GPT-4) to generate a list of required documents and instructions on how to create them based on the user's input, and provides them to the user.

[1525] Step 7:

[1526] The user acts on the information provided.

[1527] Input: Information provided by the server

[1528] Output: User actions

[1529] Specific operations: Based on the information provided by the server, the user purchases products, participates in events, applies for visas, etc.

[1530] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1531] "Example 1"

[1532] As one embodiment of the present invention, a system incorporating an emotion engine is provided. This system analyzes emotions from a user's text input or voice input. Specifically, when a user composes an email, the system analyzes the user's emotional state from the content of the email. For example, if the user uses the expression "I'm in trouble," the system analyzes that the user is facing a difficult situation.

[1533] "Example 2"

[1534] Based on the analyzed emotions, the system can tailor its responses and services to the user. For example, if the system detects that the user is facing a difficult situation, it can provide more specific assistance to the user, such as providing detailed guidance on the visa application process or providing additional information the user may need.

[1535] "Example 3"

[1536] Furthermore, the emotion engine works in conjunction with other functions of the system. For example, the email summary function adjusts the content of the summary by taking into account the user's emotional state analyzed by the emotion engine. For example, if the user expresses anger, the summary function will highlight that information. This allows the user to understand their own emotional state more clearly and take appropriate action.

[1537] The process flow of each embodiment will be described below.

[1538] "Example 1"

[1539] Step 1: The user enters text or voice input into the system.

[1540] Step 2: The sentiment engine analyzes the user's input for sentiment, taking into account word choice, context, tone of voice, etc.

[1541] Step 3: The analysis results are fed back to other functions of the system, allowing it to respond according to the user's emotional state.

[1542] "Example 2"

[1543] Step 1: The emotion engine analyzes the user's emotions.

[1544] Step 2: Based on the analysis results, the system tailors its response and services to the user.

[1545] Step 3: For example, if the system analyzes that the user is facing a difficult situation, it will provide more specific assistance to the user.

[1546] "Example 3"

[1547] Step 1: The emotion engine analyzes the user's emotions.

[1548] Step 2: The analysis results are fed back to other functions of the system, such as the email digest function.

[1549] Step 3: The email summary feature adjusts the summary content taking into account the user's emotional state as analyzed by the sentiment engine.

[1550] Example 1

[1551] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1552] In today's business environment, there is a demand for efficient processing of multiple tasks, such as summarizing emails, handling multiple languages, and supporting visa applications. However, processing these tasks individually requires a lot of time and effort, which places a heavy burden on users. It is also important to understand the user's situation through sentiment analysis and respond appropriately, but current systems do not integrate these functions, making it difficult to respond efficiently.

[1553] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1554] In this invention, the server includes a means for summarizing emails, a means for converting keywords in the user's native language into Japanese sentences, a means for summarizing advance materials into the user's native language, a means for instantly summarizing speeches of participants, a means for instantly researching difficult words, a means for informing the user of what day it is and what customs exist, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules and for supporting the preparation of documents required for applications, a means for analyzing emotions from the user's text input or voice input, and a means for generating Japanese sentences using a generative AI model. This makes it possible to efficiently process multiple tasks, reduce the user's burden, and enable appropriate responses.

[1555] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[1556] The "means for converting keywords in one's native language into sentences in Japanese" refers to a means for analyzing keywords in one's native language input by a user and generating corresponding sentences in Japanese.

[1557] The "means for summarizing advance materials into the native language" refers to a means for analyzing materials provided in advance and generating a summary in the native language.

[1558] The "means for instantly summarizing participant speech" refers to a means for summarizing in real time what is said by participants in a meeting or event.

[1559] An "instant lookup tool for difficult words" is a tool that enables users to instantly look up words or technical terms that are difficult for them to understand and provide their meanings and explanations.

[1560] A "means of teaching what day and customs it is today" is a means of providing information about events and customs associated with a particular date.

[1561] The "Visa Application and Renewal Support Tool" is a tool that generates guidance necessary for visa applications and renewals based on the latest visa and immigration rules, and supports the preparation of documents necessary for applications.

[1562] An "emotion analysis means" is a means for analyzing emotions from a user's text input or voice input and identifying the user's emotional state.

[1563] A "means for generating Japanese sentences using a generative AI model" is a means for generating Japanese sentences based on input data using a generative AI model.

[1564] This invention is a system including a means for summarizing emails, a means for converting keywords in one's native language into Japanese sentences, a means for summarizing advance materials into one's native language, a means for instantly summarizing speech by participants, a means for instantly researching difficult words, a means for informing participants what day it is and what customs exist, a means for supporting visa applications and renewals, a means for sentiment analysis, and a means for generating Japanese sentences using a generative AI model.

[1565] Email summary method

[1566] The server receives the email body and analyzes it using a natural language processing library (e.g. NLTK, spaCy). It extracts important keywords and phrases from the analyzed text and generates a summary. The summary is sent to the terminal and displayed to the user.

[1567] Examples:

[1568] A user sends an email saying, "I am sending you the minutes of the meeting. Key points are listed below."

[1569] The server extracts the keywords "meeting," "minutes," and "important points" and generates a summary sentence such as "We will send you the important points of the meeting minutes."

[1570] The terminal displays the summary to the user.

[1571] A means of converting keywords in your native language into Japanese sentences

[1572] The user enters a keyword in their native language into the device, which then sends the keyword to the server. The server uses a generative AI model (e.g. GPT-3) to analyze the keyword and generate a corresponding Japanese sentence. The generated Japanese sentence is sent to the device and displayed to the user.

[1573] Examples:

[1574] A user enters the keyword "meeting."

[1575] The server generates the Japanese sentence "conference" and sends it to the terminal.

[1576] The terminal displays Japanese text to the user.

[1577] Emotion analysis means

[1578] The user inputs text or voice into the device, and the device sends the input data to the server. The server uses an emotion analysis engine (e.g. IBM Watson, Microsoft Azure Text Analytics) to analyze the user's emotions and identify their emotional state. The analysis results are sent to the device and displayed to the user.

[1579] Examples:

[1580] The user enters the text "I'm having trouble."

[1581] The server analyzes the expression "I'm in trouble" and determines that the user is facing a difficult situation.

[1582] The terminal displays the analysis results to the user.

[1583] Examples of prompt statements

[1584] Email summary prompt:

[1585] Enter the body of your email and we'll extract the key information and generate a summary.

[1586] Prompt sentence for converting keywords in your native language into Japanese sentences:

[1587] Please enter a keyword in your native language and we will generate the corresponding Japanese sentence.

[1588] Emotion engine prompt:

[1589] Enter text or voice and we'll analyze the sentiment from your input.

[1590] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1591] Email summary method

[1592] Step 1:

[1593] A user sends an email.

[1594] Input: Email body

[1595] Output: Emails stored on the mail server

[1596] Specific behavior: A user composes an email using an email client and presses the send button.

[1597] Step 2:

[1598] The server receives the email from the mail server.

[1599] Input: Emails stored on the mail server

[1600] Output: The email body forwarded to the server

[1601] Specific operation: The server periodically checks the mail server and receives new mail.

[1602] Step 3:

[1603] The server tokenizes the email body using a natural language processing library (e.g. NLTK, spaCy).

[1604] Input: Email body

[1605] Output: A tokenized word list

[1606] Specific operation: The server analyzes the email body and splits it into words.

[1607] Step 4:

[1608] The server tags the tokenized words with parts of speech and extracts important keywords and phrases.

[1609] Input: A tokenized word list

[1610] Output: A list of important keywords and phrases

[1611] What it does: The server tags each word with a part-of-speech tag and applies an algorithm to rate its importance.

[1612] Step 5:

[1613] A summary is generated based on the keywords and phrases extracted by the server.

[1614] Input: A list of important keywords and phrases

[1615] Output: Summary

[1616] What it does: The server uses a summary generation algorithm to create a short sentence containing the key information.

[1617] Step 6:

[1618] The server transmits the generated summary to the terminal.

[1619] Input: Abstract

[1620] Output: Summary sent to terminal

[1621] Specific operation: The server transmits the summary text to the terminal via the network.

[1622] Step 7:

[1623] The terminal displays the summary to the user.

[1624] Input: Abstract

[1625] Output: Summary displayed to the user

[1626] Specific operation: The terminal displays the summary on the display.

[1627] A means of converting keywords in your native language into Japanese sentences

[1628] Step 1:

[1629] A user inputs a keyword in his / her native language into a terminal.

[1630] Input: Keywords in your native language

[1631] Output: Keywords typed into the terminal

[1632] Specific Action: A user enters a keyword using a keyboard or voice input.

[1633] Step 2:

[1634] The terminal transmits the entered keyword to the server.

[1635] Input: Keywords in your native language

[1636] Output: Keywords sent to the server

[1637] Specific operation: The terminal sends the keyword to the server via the network.

[1638] Step 3:

[1639] The server analyzes the keywords and generates corresponding Japanese sentences.

[1640] Input: Keywords in your native language

[1641] Output: Japanese text

[1642] Specific operation: The server uses a generative AI model (e.g. GPT-3) to generate Japanese sentences based on keywords.

[1643] Step 4:

[1644] The Japanese text generated by the server is sent to the terminal.

[1645] Input: Japanese text

[1646] Output: Japanese text sent to the terminal

[1647] Specific operation: The server sends Japanese text to the terminal via the network.

[1648] Step 5:

[1649] The terminal displays Japanese text to the user.

[1650] Input: Japanese text

[1651] Output: Japanese text displayed to the user

[1652] Specific operation: The device displays Japanese text on the display.

[1653] Emotion analysis means

[1654] Step 1:

[1655] The user inputs text or voice into the terminal.

[1656] Input: Text or audio data

[1657] Output: Text or audio data entered into the device

[1658] Specific action: A user inputs text or voice using a keyboard or microphone.

[1659] Step 2:

[1660] The terminal transmits the entered text and voice data to the server.

[1661] Input: Text or audio data

[1662] Output: The text or audio data sent to the server.

[1663] Specific operation: The device sends text and voice data to the server via the network.

[1664] Step 3:

[1665] The server analyzes the user's emotions using a sentiment analysis engine (e.g., IBM Watson, Microsoft Azure Text Analytics).

[1666] Input: Text or audio data

[1667] Output: Emotion analysis results

[1668] What happens: The server uses a sentiment analysis engine to identify sentiment from the input data.

[1669] Step 4:

[1670] The server sends the analysis results to the terminal.

[1671] Input: Sentiment analysis results

[1672] Output: Analysis results sent to the device

[1673] Specific operation: The server transmits the analysis results to the terminal via the network.

[1674] Step 5:

[1675] The terminal displays the analysis results to the user.

[1676] Input: Sentiment analysis results

[1677] Output: Analysis results displayed to the user

[1678] Specific operation: The device displays the analysis results on the display.

[1679] (Application example 1)

[1680] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1681] In today's information society, users receive large amounts of emails and messages and are required to manage them efficiently. It is also important to provide appropriate content according to the user's emotional state. However, with conventional systems, it was difficult to perform email summarization, emotional analysis, and content recommendation according to emotions in a unified manner. This often caused users to feel stressed by information overload, making it difficult to manage information efficiently.

[1682] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes an email summarizing means, a means for converting keywords in the native language into Japanese sentences, an emotion analyzing means, and a means for recommending content according to the emotional state. This allows the user to efficiently summarize received emails and messages and receive appropriate content according to the emotional state.

[1683] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[1684] The "means for converting keywords in one's native language into sentences in Japanese" refers to a means for analyzing keywords in one's native language input by a user and generating corresponding sentences in Japanese.

[1685] The "means for summarizing advance materials into the native language" refers to a means for analyzing materials provided in advance and generating a summary in the native language.

[1686] The "means for instantly summarizing participant speech" refers to a means for summarizing in real time what is said by participants in a meeting or event.

[1687] An "instant lookup tool for difficult words" is a tool that allows users to instantly look up words or technical terms that are difficult for them to understand and provide their meanings.

[1688] A "means of telling what day it is and what customs it has" is a means of providing information about events and customs related to the current date.

[1689] "Means of generating appropriate guidance regarding visa applications and renewals and supporting the preparation of documents required for applications" refers to means of providing guidance necessary for visa applications and renewals and supporting the preparation of necessary documents based on the latest visa and immigration rules.

[1690] An "emotion analysis means" is a means for analyzing emotions from a user's text input or voice input.

[1691] The "means for recommending content according to an emotional state" refers to a means for recommending appropriate content based on an analyzed emotional state of a user.

[1692] The following system configuration will be described as an embodiment of the present invention.

[1693] System Configuration

[1694] This system consists of a server and a user terminal. The server includes a means for summarizing emails, a means for converting keywords in the user's native language into Japanese sentences, a means for sentiment analysis, and a means for recommending content according to the user's emotional state. The user terminal is a device such as a smartphone, tablet, or PC, and provides an interface for users to receive emails and messages and use various functions of the system.

[1695] Hardware and Software Used

[1696] Hardware: Smartphones, tablets, computers

[1697] Software: Python, OpenAI API, TextBlob

[1698] Data processing and calculation

[1699] 1. Email summary methods:

[1700] The server analyzes the text of the email received by the user, extracts important information, and generates a summary. For this process, it uses OpenAI's GPT-3 model. Specifically, it inputs the text of the email as a prompt and generates a summary.

[1701] 2. How to convert keywords in your native language into Japanese sentences:

[1702] The server analyzes the native language keywords entered by the user and generates corresponding Japanese sentences. For this process, OpenAI's GPT-3 model is used. The keyword is entered as a prompt and Japanese sentences are generated.

[1703] 3. Emotion analysis means:

[1704] The server analyzes the sentiment from the user's text and voice input using the TextBlob library. It calculates the optimism (positive / negative) and subjectivity of the text.

[1705] 4. How to recommend content based on your emotional state:

[1706] The server recommends appropriate content based on the analyzed emotional state of the user, for example, recommending relaxing content if the user shows negative emotions.

[1707] Examples

[1708] Email summary: The email body "Please prepare for tomorrow's meeting" is summarized to "Please prepare for the meeting."

[1709] Sentiment analysis: Analyze negative emotions from the text "I'm in trouble" and recommend content to help the user relax.

[1710] Keyword conversion: Convert the keyword "meeting" to "I have a meeting."

[1711] Examples of prompt statements

[1712] Email Summary: "Please summarize the email body below:\n\n{email body}"

[1713] Keyword conversion: "Convert the following keyword into a Japanese sentence: {keyword}"

[1714] In this way, users can efficiently manage their received emails and messages and receive appropriate content according to their emotional state.

[1715] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1716] Step 1:

[1717] A user receives an email on a terminal. The terminal sends the body of the received email to a server. The input is the email body, and the output is sending the email body to the server.

[1718] Step 2:

[1719] The server analyzes the received email body, extracts important information, and generates a summary. Specifically, it uses OpenAI's GPT-3 model to generate a summary sentence by inputting the email body as a prompt. The input is the email body, and the output is the summary sentence.

[1720] Step 3:

[1721] The server sends the generated summary to the terminal, which displays the summary to the user. The input is the summary, and the output is the display of the summary to the user.

[1722] Step 4:

[1723] A user inputs a keyword in his / her native language into a terminal. The terminal transmits the input keyword to a server. The input is the keyword in the native language, and the output is the transmission of the keyword to the server.

[1724] Step 5:

[1725] The server analyzes the received keywords and generates corresponding Japanese sentences. Specifically, it uses OpenAI's GPT-3 model to input the keywords as prompts and generate Japanese sentences. The input is the native language keyword, and the output is the Japanese sentence.

[1726] Step 6:

[1727] The server sends the generated Japanese text to the terminal. The terminal displays the Japanese text to the user. The input is the Japanese text, and the output is the display of the Japanese text to the user.

[1728] Step 7:

[1729] A user inputs text or voice into a terminal. The terminal sends the input text or voice data to a server. The input is text or voice data, and the output is the transmission of data to a server.

[1730] Step 8:

[1731] The server analyzes the sentiment from the received text or audio data. Specifically, it uses the TextBlob library to calculate the optimism (positive / negative) and subjectivity of the text. The input is the text or audio data, and the output is the sentiment analysis result.

[1732] Step 9:

[1733] The server recommends appropriate content based on the analyzed emotional state. For example, if the user shows negative emotions, it recommends relaxing content. The input is the emotion analysis result, and the output is the recommended content.

[1734] Step 10:

[1735] The server sends the recommended content to the terminal. The terminal displays the recommended content to the user. The input is the recommended content, and the output is the display of the content to the user.

[1736] Example 2

[1737] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[1738] In today's business environment, there is a need to quickly and accurately grasp the content of meetings and documents. However, understanding the content of documents and meetings provided in different languages ​​is a major challenge, especially for multinational companies. It is also important to provide appropriate responses and services based on the user's emotions, but there are only a limited number of systems that can do this automatically. Furthermore, the procedures for visa applications and renewals are complicated, making it difficult for users to receive appropriate guidance.

[1739] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for summarizing emails, a means for converting keywords in the native language into Japanese sentences, a means for summarizing advance materials into the native language, a means for instantly summarizing the speech of participants, a means for instantly researching difficult words, a means for informing what day it is today and what customs exist, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules, etc., and for supporting the preparation of documents required for application, and a means for adjusting responses and services to users based on emotion analysis. This enables users to quickly understand materials and meeting contents in different languages ​​and receive appropriate guidance and services.

[1740] The "email summarization means" is a means for analyzing the body of an e-mail, extracting important information, and generating a concise summary.

[1741] The "means for converting keywords in a user's native language into Japanese sentences" refers to a means for converting keywords entered in a user's native language into Japanese sentences.

[1742] The "means for summarizing advance materials into the user's native language" refers to a means for analyzing materials distributed in advance and generating a summary of the materials in the user's native language.

[1743] The "means for instantly summarizing participant speech" refers to a means for analyzing speech during a conference in real time and generating a summary thereof.

[1744] An "instant lookup tool for difficult words" is a tool that instantly looks up difficult words and technical terms and provides their meanings and explanations.

[1745] A "means of teaching what day and customs it is today" is a means of providing information about events and customs associated with a particular date.

[1746] "Means of generating appropriate guidance regarding visa applications and renewals and supporting the preparation of documents required for applications" means means of providing appropriate guidance regarding visa applications and renewals and supporting the preparation of necessary documents based on the latest visa and immigration rules.

[1747] "Means for adjusting responses and services to users based on emotion analysis" refers to means for analyzing the emotions of users and providing appropriate responses and services based on the results.

[1748] The present invention provides a system that enables a user to quickly understand materials and meeting contents in different languages ​​and receive appropriate guidance and services. A specific embodiment of this system will be described below.

[1749] 1. Email Summary

[1750] The server analyzes the body of the email, extracts important information, and generates a concise summary. The software used includes APIs that provide natural language processing techniques. For example, a natural language processing API is used to analyze the content of the email and extract key points.

[1751] Examples:

[1752] The server analyzes the long email received by the user and generates a summary. For example, "Please summarize the following email."

[1753] 2. A way to convert keywords in your native language into Japanese sentences

[1754] The server converts the keywords entered in the user's native language into Japanese text. The software used includes a translation API. For example, the server converts the keywords into Japanese text using a translation API.

[1755] Examples:

[1756] When a user enters "meeting" in English, the server converts it to "会議" in Japanese. For example, "Please convert the following keywords into Japanese."

[1757] 3. Summary means for pre-distributed materials into the native language

[1758] The server analyzes the pre-distributed materials and generates a summary in the user's native language. The software used includes natural language processing technology and translation APIs. For example, use the natural language processing API to analyze the content of the materials, extract important points, and use the translation API to translate the summary into the native language.

[1759] Specific example:

[1760] The user uploads English meeting minutes, and the server analyzes the minutes to generate a Japanese summary. For example, "Please summarize the following English meeting minutes in Japanese."

[1761] 4. Real-time summary means for participants' speeches

[1762] The server analyzes the speeches during the meeting in real time and generates a summary. The hardware used includes a microphone, and the software includes a speech recognition API and a natural language processing API. For example, use the speech recognition API to convert the speech into text, and use the natural language processing API to generate a summary.

[1763] Specific example:

[1764] Convert the English speeches during the meeting into text in real time, analyze the text, and generate a Japanese summary. For example, "Please convert the following voice data into text and summarize it in Japanese."

[1765] 5. Real-time investigation means for difficult words

[1766] The server instantly searches difficult words and technical terms and provides their meanings and explanations. The software used includes a dictionary API and a knowledge base. For example, the dictionary API is used to get the meaning of difficult words.

[1767] Examples:

[1768] If a user types in the word "computer vision," the server looks up the meaning and provides an explanation, e.g., "Please look up the meaning of the following words."

[1769] 6. A way to tell people what day it is and what customs apply

[1770] The server provides information about events and customs associated with a particular date. The software used may include a calendar API and an event database. For example, the calendar API may be used to retrieve event information for a particular date.

[1771] Examples:

[1772] When a user asks "What day is it today?", the server provides the events and customs of that day. For example, "Tell me what day it is today."

[1773] 7. A means to generate appropriate guidance on visa applications and renewals and to assist in preparing the necessary documents for application

[1774] The server provides appropriate guidance for visa application and renewal based on the latest visa and immigration rules and assists in preparing the necessary documents. The software used includes a regulatory database and a guidance generation engine. For example, the server uses the regulatory database to obtain the latest rules and provides appropriate guidance using the guidance generation engine.

[1775] Examples:

[1776] If a user types in "What is the procedure for applying for a visa?", the server will provide detailed guidance based on the latest rules. For example, "What is the procedure for applying for a visa?"

[1777] 8. Means of tailoring responses and services to users based on sentiment analysis

[1778] The server analyzes the user's emotions and provides appropriate responses and services based on the results. The software used includes an emotion analysis API, for example, to understand the user's emotions and respond accordingly.

[1779] Examples:

[1780] If a user types "It's difficult to apply for a visa," the server can analyze the sentiment and provide specific procedural guidance. For example, "Please analyze the user's sentiment from the following text and generate an appropriate response."

[1781] This system enables users to quickly understand materials and meeting content in different languages ​​and receive appropriate guidance and services.

[1782] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1783] Step 1:

[1784] Users upload pre-distributed materials to the system. The input is in the form of a PDF file or a Word document. The output is the material data stored on the server. The specific operation is that the user selects a file through the system interface and clicks the upload button.

[1785] Step 2:

[1786] The server receives the uploaded materials and uses natural language processing technology to analyze the contents of the materials. The input is the uploaded material data, and the output is the key points of the analyzed materials. Specifically, the server calls the natural language processing API to analyze the contents of the materials.

[1787] Step 3:

[1788] The server generates a summary of the material based on the analysis results. The input is the key points of the analyzed material, and the output is the generated summary. Specifically, the server uses the generative AI model to generate the summary.

[1789] Step 4:

[1790] The server translates the generated summary into the user's native language. The input is the generated summary, and the output is the translated summary. In concrete terms, the server calls a translation API to translate the summary into the user's native language.

[1791] Step 5:

[1792] The terminal captures speech during a conference through a microphone. The input is voice data, and the output is the captured voice data. In a specific operation, the terminal uses the microphone to record the voice.

[1793] Step 6:

[1794] The server converts the acquired voice data into text using voice recognition technology. The input is the acquired voice data, and the output is the text of the spoken content. In concrete terms, the server calls the voice recognition API and converts the voice data into text.

[1795] Step 7:

[1796] The server analyzes the textualized utterance. The input is the textualized utterance, and the output is the key points of the analyzed utterance. Specifically, the server analyzes the utterance using a natural language processing API.

[1797] Step 8:

[1798] The server generates a summary of the utterance based on the analysis results. The input is the key points of the analyzed utterance, and the output is the generated summary. Specifically, the server generates the summary using the generative AI model.

[1799] Step 9:

[1800] The server receives the user's input data and analyzes the emotions using emotion analysis technology. The input is the user's text or voice data, and the output is the analyzed emotional state. Specifically, the server calls the emotion analysis API and analyzes the user's emotions.

[1801] Step 10:

[1802] The server adjusts responses and services to the user based on the analysis results. The input is the analyzed emotional state, and the output is the adjusted response or service. The specific operation is for the server to perform processing to provide appropriate guidance and information.

[1803] Step 11:

[1804] The server provides appropriate guidance for visa application and renewal based on the latest visa and immigration rules, and assists in preparing the necessary documents. The input is the user's questions and information on the visa application, and the output is detailed guidance and a list of documents. In concrete terms, the server consults the regulatory database and provides appropriate guidance using a guidance generation engine.

[1805] (Application example 2)

[1806] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1807] In today's business environment, making meetings and conferences more efficient is an important issue. In particular, in meetings and conferences held inside self-driving vehicles, there is a need to efficiently share information and make decisions while on the move. However, there is a lack of means to summarize the contents of meetings in real time and provide them to participants. It is also difficult to analyze participants' emotions and provide appropriate responses and services. To solve these issues, it is necessary to analyze utterances in real time, generate summaries, and adjust responses based on emotion analysis.

[1808] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1809] In this invention, the server includes a means for summarizing emails, a means for converting keywords in the native language into Japanese sentences, a means for summarizing advance materials into the native language, a means for instantly summarizing participants' speech, a means for instantly researching difficult words, a means for informing participants of what day it is and what customs exist, a means for generating appropriate guidance for visa applications and renewals in accordance with the latest visa and immigration rules, etc., and for supporting the preparation of documents required for applications, a means for analyzing speeches of meetings and conferences held in the vehicle in real time, generating summaries and displaying them on an in-vehicle display or smart glasses, and a means for adjusting responses and services to stress and difficult situations while driving based on the analyzed emotions. This enables meetings and conferences in self-driving vehicles to be made more efficient, reduces stress for participants, and provides appropriate responses and services.

[1810] The "email summarization means" is a means for analyzing the body of an e-mail, extracting important information, and generating a summary.

[1811] The "means for converting keywords in a user's native language into Japanese sentences" refers to a means for converting keywords entered in a user's native language into Japanese sentences.

[1812] The "means for summarizing advance materials into the user's native language" refers to a means for analyzing materials distributed in advance and generating a summary of the materials in the user's native language.

[1813] The "means for instantly summarizing participant speech" refers to a means for analyzing speech during a conference in real time and generating a summary thereof.

[1814] An "instant lookup means for difficult words" is a means for instantly looking up difficult words or technical terms that come up during a meeting and providing their meanings.

[1815] A "means of telling what day it is and what customs it has" is a means of providing information about events and customs related to the current date.

[1816] "Means of generating appropriate guidance on visa applications and renewals and supporting the preparation of documents required for applications" means means of providing guidance on visa applications and renewals and supporting the preparation of necessary documents based on the latest visa and immigration rules.

[1817] "A means for analyzing in-vehicle conference or meeting speech in real time, generating a summary, and displaying it on an in-vehicle display or smart glasses" refers to a means for analyzing in-vehicle conference or meeting speech in real time, generating a summary, and displaying it on an in-vehicle display or smart glasses.

[1818] "Means for adjusting responses and services to stressful or difficult situations while driving, based on analyzed emotions" refers to means for analyzing emotions from speech and behavior during a meeting, and providing appropriate responses and services according to stressful or difficult situations while driving.

[1819] A system for implementing the present invention uses the following hardware and software.

[1820] Hardware used:

[1821] In-car display

[1822] Smart Glasses

[1823] microphone

[1824] High-performance processor (e.g. NVIDIA Jetson)

[1825] Software used:

[1826] Voice recognition software (e.g. Google Cloud Speech-to-Text)

[1827] Natural language processing libraries (e.g. spaCy, NLTK)

[1828] Sentiment analysis models (e.g. BERT, GPT-3)

[1829] Summary generation models (e.g. T5, BART)

[1830] Overview of what the system does:

[1831] The server first collects speech from meetings and conferences using the in-car microphone. The collected voice data is converted into text data using Google Cloud Speech-to-Text. This text data is then analyzed using a natural language processing library (spaCy or NLTK) to extract important keywords and phrases. A summary is generated based on the extracted keywords and phrases using a summary generation model (T5 or BART). This summary is then displayed on the in-car display or smart glasses.

[1832] In addition, the server inputs the generated text data into a sentiment analysis model (BERT or GPT-3) to analyze the user's emotions. Based on the analysis results, the server adjusts responses and services to stressful or difficult situations while driving. For example, if the user is feeling stressed, the server can play relaxing music.

[1833] Examples:

[1834] If someone says during a meeting, "I'd like to check the progress of the next project," the server will generate a "summary of the project's progress" and display it on the in-car display or smart glasses.

[1835] If the user says "this procedure is difficult," the server will provide "detailed guidance on the procedure."

[1836] Example prompt for a generative AI model:

[1837] Please summarize what was said at the meeting.

[1838] "Analyze the sentiment from user utterances and generate appropriate responses."

[1839] In this way, meetings and consultations inside self-driving vehicles can be made more efficient and stress-free for users.

[1840] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1841] Step 1:

[1842] The server collects speech from meetings and conferences using a microphone in the car. The input is voice data, and the output is the collected voice data. This voice data is used in subsequent processing steps.

[1843] Step 2:

[1844] The server converts the collected voice data into text data using Google Cloud Speech-to-Text. The input is voice data, and the output is text data. This conversion allows the voice information to be treated as text information.

[1845] Step 3:

[1846] The server analyzes the generated text data using natural language processing libraries (spaCy or NLTK) to extract important keywords and phrases. The input is the text data, and the output is the extracted keywords and phrases. This analysis makes the main points of the meeting clear.

[1847] Step 4:

[1848] The server generates a summary based on the extracted keywords and phrases using a summary generation model (T5 or BART). The input is the keywords and phrases, and the output is the summary text. This summary is a concise summary of the contents of the meeting.

[1849] Step 5:

[1850] The server displays the generated summary text on an in-car display or smart glasses. The input is the summary text, and the output is the displayed summary information, allowing users to visually confirm the main points of the meeting.

[1851] Step 6:

[1852] The server inputs the generated text data into a sentiment analysis model (BERT or GPT-3) to analyze the user's sentiment. The input is text data, and the output is the sentiment analysis result. This analysis allows the user's emotional state to be understood.

[1853] Step 7:

[1854] The server tailors responses and services to stressful or difficult situations while driving based on the results of emotion analysis. The input is the emotion analysis result, and the output is the tailored response or service. For example, if the user is feeling stressed, the server plays relaxing music.

[1855] In this way, through a series of processes, the server can make meetings and discussions inside self-driving vehicles more efficient and reduce stress for users.

[1856] Example 3

[1857] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[1858] In today's information society, users need to efficiently process and understand a lot of information. However, there is a lot of information that users cannot immediately understand, such as technical terms, abbreviations, holidays, anniversaries, and information on visa applications. There is also a demand for information that takes into account the user's emotional state and for email summaries, but there is no system that provides these in a unified manner. This means that users spend a lot of time and effort searching for and understanding information, making efficient information processing difficult.

[1859] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1860] In this invention, the server includes a means for summarizing emails, a means for converting keywords in the user's native language into Japanese sentences, a means for summarizing advance materials into the user's native language, a means for instantly summarizing speeches of participants, a means for instantly researching difficult words, a means for informing the user of what day it is and what customs exist, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules and for supporting the preparation of documents required for applications, a means for analyzing the user's emotional state and adjusting the summary content of emails based on that information, a means for analyzing words that the user does not understand, researching their meanings and uses from information sources on the Internet and providing the results, and a means for obtaining information about today's date and providing information about applicable holidays and anniversaries. This enables the user to efficiently process and understand a wide range of information.

[1861] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[1862] The "means for converting native language keywords into Japanese sentences" refers to a means for analyzing native language keywords entered by a user and converting them into appropriate Japanese sentences.

[1863] The "means for summarizing advance materials into the user's native language" refers to a means for analyzing materials provided in advance and summarizing their contents in the user's native language.

[1864] The "means for instantly summarizing participant utterances" refers to a means for analyzing in real time the contents of utterances made by participants in a meeting or conversation, and generating a summary.

[1865] An "instant lookup tool for difficult words" is a tool that analyzes technical terms and abbreviations that a user does not understand, looks up their meaning and usage from online sources, and provides the results.

[1866] "A means of informing people of what day it is and what customs apply today" is a means of obtaining information about today's date and providing information about the corresponding national holidays, anniversaries, and customs and events that take place on that day.

[1867] "Means for generating appropriate guidance regarding visa applications and renewals, and supporting the preparation of documents required for applications" refers to means for analyzing information input by users based on the latest visa and immigration rules, generating a list of required documents, and providing instructions on how to prepare each document.

[1868] The "means for analyzing the emotional state of a user and adjusting the summary content of an email based on that information" refers to a means for analyzing the emotional state of a user and adjusting the summary content of an email based on the results of that analysis.

[1869] "A means for analyzing words that a user does not understand, researching their meaning and use from information sources on the Internet, and providing the results" refers to a means for analyzing words that a user inputs that the user does not understand, researching their meaning and use from information sources on the Internet, and providing the results.

[1870] "Means for obtaining information about today's date and providing information about corresponding holidays and anniversaries" refers to means for obtaining today's date and providing information about corresponding holidays and anniversaries.

[1871] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1872] The present invention is a system for supporting a user in efficiently processing and understanding a wide range of information. A specific embodiment of this system will be described below.

[1873] Terminology and acronym analysis and research

[1874] The server analyzes words that the user does not understand and investigates their meaning and usage. Specifically, it uses Python to utilize natural language processing libraries (e.g., NLTK and spaCy) to analyze the input words. Based on the analysis results, it retrieves related information from Internet sources (e.g., Wikipedia API and Google Search API) and provides it to the user.

[1875] Examples:

[1876] If a user types in "What is an API?", the server will parse the phrase and use the Wikipedia API to get information about the definition and uses of "API" and display it to the user.

[1877] Example prompt:

[1878] "What is an API?"

[1879] Survey of Japanese holidays and anniversaries

[1880] The server checks what day it is and provides the user with information about customs and events that take place on that day. Specifically, it uses Python to refer to a calendar API (e.g., Google Calendar API) or a public holiday database to obtain the relevant information.

[1881] Examples:

[1882] If a user asks "What day is it today?", the server uses the Google Calendar API to find out that today is Respect for the Aged Day and provides the user with information about its background and related events.

[1883] Example prompt:

[1884] "What day is it today?"

[1885] Visa Application and Renewal Guidance

[1886] The server generates a list of documents required for the visa application based on the information input by the user and provides instructions on how to prepare each document. Specifically, it uses Python to analyze the information input by the user and generates a list of documents required based on the pre-prepared visa application guidelines.

[1887] Examples:

[1888] If a user enters, "I would like to apply for a tourist visa," the server analyzes that information, generates a list of documents required for the tourist visa (e.g., passport copies, application form, photographs, etc.), and provides instructions on how to prepare each document and where to submit it.

[1889] Example prompt:

[1890] "What do I need to apply for a tourist visa?"

[1891] Integration of emotion engine and email summarization function

[1892] The server uses an emotion engine to analyze the user's emotional state and adjusts the email summary content based on that information. Specifically, it uses Python to leverage emotion analysis libraries (e.g. TextBlob and VADER) to analyze the user's emotional state. Based on the results, it adjusts the email summary algorithm and highlights important information.

[1893] Examples:

[1894] If the user indicates anger, the server analyzes the emotional state and adjusts the email digest function to highlight anger-related information.

[1895] Example prompt:

[1896] Please give me a summary of this email.

[1897] The above is a specific embodiment of this system. This enables the user to efficiently process and understand a wide range of information. The flow of the specific process in the third embodiment will be described with reference to FIG.

[1898] Terminology and acronym analysis and research

[1899] Step 1:

[1900] The user types words they do not understand.

[1901] Input: "What is an API?"

[1902] Output: The input word "API"

[1903] Step 2:

[1904] The server analyzes the entered words.

[1905] Input: The entered word "API"

[1906] Data processing: Use Python to analyze words using natural language processing libraries (e.g. NLTK and spaCy).

[1907] Output: Analysis results (e.g. "API stands for Application Programming Interface")

[1908] Step 3:

[1909] The server retrieves relevant information from sources on the Internet.

[1910] Input: Analysis result (e.g. "API stands for Application Programming Interface")

[1911] Data computation: Uses the Wikipedia API to obtain information about the definition and uses of "API."

[1912] Output: The information obtained (e.g. "An API provides an interface between software programs")

[1913] Step 4:

[1914] The server provides the obtained information to the user.

[1915] Input: The information obtained (e.g., "An API provides an interface between software programs.")

[1916] Output: Information displayed to the user (e.g., "An API provides an interface between software programs")

[1917] Survey of Japanese holidays and anniversaries

[1918] Step 1:

[1919] A user enters a question about today's date.

[1920] Input: "What day is it today?"

[1921] Output: Given the question "What day is it today?"

[1922] Step 2:

[1923] The server gets today's date.

[1924] Input: Current date

[1925] Data processing: Get the current date using Python.

[1926] Output: Retrieved date (e.g. September 21st)

[1927] Step 3:

[1928] The server consults the holiday database.

[1929] Input: Acquired date (e.g. September 21st)

[1930] Data calculation: Refer to the Google Calendar API or the holiday database to check the applicable holidays and anniversaries.

[1931] Output: Information about holidays and anniversaries (e.g. "Today is Respect for the Aged Day")

[1932] Step 4:

[1933] The server provides the user with information about applicable holidays and anniversaries.

[1934] Input: Holiday or anniversary information (e.g. "Today is Respect for the Aged Day")

[1935] Output: Information displayed to the user (e.g. "Today is Respect for the Aged Day. Respect for the Aged Day is a day to honor the elderly and is typically celebrated with family.")

[1936] Visa Application and Renewal Guidance

[1937] Step 1:

[1938] The user enters information about their visa application.

[1939] Input: "I want to apply for a tourist visa"

[1940] Output: Input information: "I would like to apply for a tourist visa."

[1941] Step 2:

[1942] The server parses the input information.

[1943] Input:Information entered: "I would like to apply for a tourist visa"

[1944] Data Processing: Use Python to analyze user input and identify documents required for a tourist visa.

[1945] Output: List of required documents (e.g. passport copies, application forms, photos)

[1946] Step 3:

[1947] The server generates a list of required documents.

[1948] Input: List of required documents (e.g. passport copies, application forms, photos)

[1949] Data calculation: Generate document list based on pre-prepared visa application guidelines.

[1950] Output: Generated document list

[1951] Step 4:

[1952] The server instructs the user on how to create each document.

[1953] Input: Generated document list

[1954] Output: Instructions to be displayed to the user (e.g. "Please copy all pages of your passport. Application forms can be downloaded from the embassy's website.")

[1955] Integration of emotion engine and email summarization function

[1956] Step 1:

[1957] A user requests a summary of their email.

[1958] Input: "What is the summary of this email?"

[1959] Output: Input request: "Please tell me the summary of this email."

[1960] Step 2:

[1961] The server analyzes the user's emotional state.

[1962] Input: The user's emotional state

[1963] Data processing: Analyze the emotional state using a sentiment analysis library (e.g. TextBlob or VADER).

[1964] Output: Analysis results (e.g., emotion of anger)

[1965] Step 3:

[1966] The server parses the email content.

[1967] Input: Email content

[1968] Data Calculation: Analyze the body of the email and extract important information.

[1969] Output: Extracted important information

[1970] Step 4:

[1971] The server generates a summary taking into account the emotional state.

[1972] Input: Analysis results (e.g., emotion of anger), extracted important information

[1973] Data Computation: Adjust the summarization algorithm to take into account the emotional state and generate summaries.

[1974] Output: Generated summary (e.g. "This email is about the progress of the project, highlighting any delays that may have occurred")

[1975] The above are the specific processing steps of the program of this system.

[1976] (Application example 3)

[1977] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1978] Modern brick-and-mortar stores are required to improve customer service, but there is a lack of systems that can instantly answer customer questions or provide information on events and promotions related to a particular day. There are also limited means to provide the latest visa information and application guidance for foreign tourists. Furthermore, there is no system that can analyze the emotional state of customers and provide appropriate information. To solve these issues, a system that integrates multiple functions is needed.

[1979] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1980] In this invention, the server includes a means for summarizing emails, a means for converting keywords in the native language into Japanese sentences, a means for summarizing advance materials into the native language, a means for instantly summarizing speeches of participants, a means for instantly researching difficult words, a means for informing a user of what day it is and what customs exist, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules and the like and for supporting the preparation of documents required for applications, a means for instantly answering customer questions, a means for providing event and promotion information related to a specific day, and a means for analyzing emotional states and providing appropriate information. This makes it possible to improve customer service, respond to foreign tourists, and provide information based on the customer's emotions.

[1981] The "email summarization means" is a means for analyzing the body of an email, extracting important information, and generating a summary.

[1982] The "means for converting keywords in a user's native language into Japanese sentences" refers to a means for converting keywords in a user's native language inputted into Japanese sentences.

[1983] "Means of summarizing advance materials into one's native language" refers to means of summarizing materials prepared in advance into one's native language.

[1984] The "means for instantly summarizing participant speech" refers to a means for instantly summarizing the contents of speech by participants in a conference or event.

[1985] An "instant lookup tool for difficult words" is a tool that enables users to instantly look up words or technical terms that they do not understand and provide their meanings and uses.

[1986] "Means of teaching what day it is and what customs exist" refers to a means of providing information about what day it is and the customs and events that take place on that day.

[1987] "A means of generating appropriate guidance regarding visa applications and renewals in accordance with the latest visa and immigration rules, etc., and supporting the preparation of documents necessary for applications" refers to a means of providing guidance necessary for visa applications and renewals based on the latest visa information and immigration rules, and supporting the preparation of necessary documents.

[1988] "Means for instantly answering customer questions" refers to a means for providing instant answers to questions asked by customers in a physical store.

[1989] The "means for providing information about events and promotions related to a specific day" refers to a means for providing customers with information about events and promotions related to a specific day.

[1990] The "means for analyzing an emotional state and providing appropriate information" is a means for analyzing an emotional state of a customer and providing appropriate information based on the result of the analysis.

[1991] The system for carrying out the present invention comprises a server and a user terminal. The server includes a means for summarizing emails, a means for converting keywords in a user's native language into Japanese sentences, a means for summarizing advance materials into the user's native language, a means for instantly summarizing speeches of participants, a means for instantly researching difficult words, a means for informing a user of what day it is and what customs exist today, a means for generating appropriate guidance on visa applications and renewals in accordance with the latest visa and immigration rules and the like, a means for supporting the preparation of documents required for applications, a means for instantly answering customer questions, a means for providing event and promotion information related to a specific day, and a means for analyzing an emotional state and providing appropriate information.

[1992] Hardware and Software Used

[1993] Hardware: Smartphones, servers

[1994] Software: Python, API request library (requests), natural language processing library (TextBlob)

[1995] Data processing and calculation

[1996] 1. Email Summarization: The server analyzes the body of an email, extracts important information, and generates a summary, allowing the user to quickly grasp the important information.

[1997] 2. Means of converting keywords in the user's native language into Japanese sentences: The server converts the keywords entered by the user in the user's native language into Japanese sentences, making it easier for foreign users to understand information in Japanese.

[1998] 3. Means for summarizing pre-prepared materials into the user's native language: The server summarizes pre-prepared materials into the user's native language, allowing the user to quickly obtain the information he or she needs in his or her native language.

[1999] 4. Means for instantly summarizing participant speech: The server instantly summarizes what participants say in meetings, events, etc. This allows users to quickly grasp the main points of the meeting.

[2000] 5. Instant search for difficult words: The server can instantly search for words or technical terms that the user does not understand and provide their meaning and usage, allowing the user to quickly understand technical information.

[2001] 6. Means of informing users of what day it is and what customs are observed: The server provides information on what day it is, the customs and events that are observed on that day, etc. This allows users to know the special information for that day.

[2002] 7. Means for generating appropriate guidance for visa application and renewal in accordance with the latest visa and immigration rules, etc., and supporting the preparation of documents required for application: The server provides guidance required for visa application and renewal based on the latest visa information and immigration rules, and supports the preparation of necessary documents. This allows foreign users to smoothly proceed with the visa application process.

[2003] 8. Means of immediate response to customer questions: The server will provide immediate answers to questions asked by customers in physical stores, thereby improving customer service.

[2004] 9. Means for Providing Event and Promotion Information Related to a Specific Day: The server provides event and promotion information related to a specific day to the customer, so that the customer can be aware of special events and promotions for that day.

[2005] 10. Means for analyzing emotional state and providing appropriate information: The server analyzes the emotional state of the customer and provides appropriate information based on the results. This allows services to be provided according to the customer's emotions.

[2006] Examples

[2007] For example, if a user asks "What documents are required for a visa application?", the server would use the following API to retrieve the information and provide the latest visa information.

[2008] Example prompt:

[2009] When a user asks "What documents do I need to apply for a visa?", use the following API to get the information to provide the latest visa information.

[2010] The flow of the specific process in the application example 3 will be described with reference to FIG.

[2011] Step 1:

[2012] A user types a question into a terminal, for example, "What documents do I need to apply for a visa?"

[2013] Input: The user's question text

[2014] Output: The question text is sent to the server.

[2015] Step 2:

[2016] The server analyzes the received question text and understands the intent of the question. It uses a natural language processing library (TextBlob) to analyze the meaning of the text.

[2017] Input: Question text

[2018] Output: Parsed question intent

[2019] Step 3:

[2020] The server generates an appropriate API request based on the parsed intent of the question, for example, to get visa information.

[2021] Input: Parsed question intent

[2022] Output: API request

[2023] Step 4:

[2024] The server sends the generated API request to an external visa information service.

[2025] Input: API request

[2026] Output: Response from the visa information service

[2027] Step 5:

[2028] The server analyzes the response received from the visa information service and extracts the necessary information.

[2029] Input: Response from the visa information service

[2030] Output: Extracted visa information

[2031] Step 6:

[2032] The server converts the extracted visa information into a format that is easy for the user to understand, for example by generating a list of required documents and providing instructions on how to complete each one.

[2033] Input: Extracted visa information

[2034] Output: Visa information for the user

[2035] Step 7:

[2036] The server transmits the generated visa information for the user to the terminal.

[2037] Input: Visa information for the user

[2038] Output: Visa information sent to the terminal

[2039] Step 8:

[2040] The terminal displays the received visa information to the user, who can then check the list of required documents and how to prepare them.

[2041] Input: Visa information sent to the terminal

[2042] Output: Visa information displayed to the user

[2043] In this way, the server provides appropriate information in response to a question entered by the user, enabling the user to quickly obtain the information he or she needs.

[2044] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2045] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2046] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[2047] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[2048] [Third embodiment]

[2049] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[2050] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[2051] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a “computer” according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32.

[2052] The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[2053] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[2054] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[2055] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[2056] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information betwee...

Claims

1. means for analyzing the pre-stored material, creating a summary of the material, and translating the summary into the user's native language; means for providing said translated summary to a user; A means for acquiring utterances during a conference; A means for converting the acquired speech into text using a speech recognition technology; means for analyzing the converted speech to generate a summary and translating the summary into the user's native language; means for providing a summary of the translated utterance to a user; A system including:

2. The system of claim 1 , wherein the summary of the material and the speech is generated using generative AI to extract key points and create a summary.

3. The system of claim 1 , wherein the translation of the material and the speech uses a translation API.

Images