system

Abstract

We provide the system. [Solution] A translation agent launched on the user's terminal collects text data and image data via a target electronic information acquisition means, A server processing means that analyzes collected text data and image data and converts them into a translatable format, A translation execution means that translates data analyzed using generative AI into multiple languages, A display control means that transmits the translated data to the user's terminal and integrates it into the display interface, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including: receiving a user utterance; adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot; encoding the prompt; and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, with the increase in international tourism, there has been a demand for services that do not make visitors who speak different languages feel a language barrier. However, many electronic information, especially the interfaces of applications and websites, are designed in a single language, which causes difficulties in information acquisition and operation for users who are not proficient in foreign languages. Furthermore, on the service provider side, the increase in manpower and costs for multilingual support has become a problem. Against this background, there is a need for a solution that solves the language barriers faced by users and at the same time reduces the burden on the service provider side.

Means for Solving the Problems

[0005] This invention solves this problem using a translation agent that operates on the user's terminal. When a user launches a specific application or website, the translation agent automatically collects text or image data. The data is analyzed by a server processing unit, and a generating AI translates it into multiple languages. Once translated, the information is transmitted to the user's terminal via a display control unit and displayed to the user in an easy-to-understand format. This process significantly reduces the man-hours and costs required for multilingual support while providing users with smooth access to information.

[0006] A "translation agent" is a program that runs on the user's device and collects and translates text and image data from specific applications and websites.

[0007] "Electronic information acquisition means" refers to a method or function for collecting text data and image data from applications and websites via a device used by a user.

[0008] "Server processing means" refers to a function or means on a server that analyzes data transmitted from a user's terminal and converts it into a format suitable for translation.

[0009] "Generative AI" is an artificial intelligence technology used to translate text and image data into multiple languages.

[0010] "Translation execution means" refers to a method or function for translating data analyzed using generative AI into a target language.

[0011] "Display control means" refers to a method or function for transmitting translated information to the user's terminal and displaying it appropriately on the interface.

[0012] "Optical character recognition technology" is a technology for analyzing and extracting character information from image data. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a tagged processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a tagged RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a tagged storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] The system of this invention was developed to improve user convenience and reduce the cost and effort of multilingual support. In this system, a translation agent runs on the user's terminal and collects and translates information from specified applications and websites to provide the user with information in their native language.

[0035] The terminal acquires screen data from applications and websites accessed by the user and sends it to the server in a specific format. The server analyzes the received data and extracts text and image data according to the format. This extracted data is translated into multiple languages ​​by a generative AI. In particular, for image data, optical character recognition technology is used as needed to extract character information and include it in the translation.

[0036] The translated data is sent from the server to the terminal and integrated into the interface in a user-friendly format by display control means. This entire process allows users to easily understand information in their native language and conduct their activities smoothly within Japan.

[0037] For example, when tourists visiting Japan use a local train transfer guidance app, they can simply open the app, and a translation agent will automatically activate, allowing them to view transfer information in their native language. This enables them to travel comfortably without feeling a language barrier, making their sightseeing even more enjoyable.

[0038] This invention is expected to not only expedite the provision of information to tourists but also significantly reduce the burden of multilingual support on public transportation and service providers within Japan. In this way, the present invention will improve the efficiency of multilingual support and contribute to building a win-win relationship between foreign visitors to Japan and Japanese tourism businesses.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The user launches the translation agent on their device and opens the application or website to be translated. The device automatically captures the content on the screen and extracts the text and image data that needs to be translated.

[0042] Step 2:

[0043] The terminal sends the extracted text and image data to the server as an analysis request. This request also includes the user's desired target language for translation.

[0044] Step 3:

[0045] The server analyzes the received data, organizing the text and extracting character information from images. Using optical character recognition (OCR) technology, it identifies characters within images and converts them into a translatable format.

[0046] Step 4:

[0047] The server uses generative AI to translate the analyzed text and image data into the specified language. During translation, adjustments are made to generate the most appropriate expression based on the context.

[0048] Step 5:

[0049] The server sends the translated data to the terminal. The terminal then uses this translation to reconfigure the application or website interface in its native language.

[0050] Step 6:

[0051] Users can view the translation results displayed on their device and obtain the necessary information. This allows them to use apps and websites without feeling any language barriers.

[0052] (Example 1)

[0053] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0054] In today's world, where multilingualism is essential, understanding information provided in different languages ​​poses a significant obstacle for users of information processing devices who are not proficient in those languages. Furthermore, information providers face the challenge of incurring substantial costs for multilingual support.

[0055] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0056] In this invention, the server includes means for a language conversion agent activated on the user's information processing device to collect textual and visual information via a target content acquisition means, data processing means for analyzing the collected textual and visual information and converting it into a language-convertible format, and conversion execution means for converting the analyzed data into multiple languages ​​using a generation AI. This allows users to easily obtain information in their native language and reduces the costs associated with multilingual support.

[0057] An "information processing device" is an electronic device that allows users to acquire and process information via the internet or applications.

[0058] A "language conversion agent" is a software program that operates on an information processing device and collects textual and visual information from content accessed by the user.

[0059] "Content acquisition means" refers to the process or technology used by an information processing device to acquire necessary data from applications or websites it accesses.

[0060] "Textual information" refers to a form of information recorded as text data, such as strings of characters found in documents and emails.

[0061] "Visual information" refers to data provided in a visual format, including elements found in images and videos.

[0062] "Data processing means" refers to processes and equipment that analyze collected textual and visual information and convert it for use in other formats or applications.

[0063] "Generative AI" is a system that uses artificial intelligence technology to generate new information or output from input data according to a specified task.

[0064] "Optical character recognition technology" is a technology that converts characters contained in visual information into electronic text data.

[0065] The system of this invention takes the form of a language conversion agent operating within an information processing device in order to improve user convenience. The information processing device acquires content from applications and websites accessed by the user and converts it into a standardized data format. In this acquisition process, text and image data are collected using screen capture technology and APIs.

[0066] The server receives data acquired from the terminal and performs analysis using its built-in data processing capabilities. During the analysis, textual information is distinguished from visual information, and the visual information is converted into text using optical character recognition technology (e.g., open-source OCR systems). This process needs to be particularly fast and accurate in applications requiring multilingual support.

[0067] The analyzed data is then translated into the specified language by the generating AI. The generating AI performs multilingual translation based on the prompt message specified during the language conversion. An example of such a prompt message is, "Translate this text into the target language."

[0068] The translated data is sent from the server to the user's information processing device, and the terminal integrates that data into the user interface using a display management mechanism. For example, when a user sightseeing in Japan uses a local guide app, the terminal automatically displays the translated content, allowing them to check information in real time without experiencing language barriers.

[0069] This allows users to easily understand information provided in different languages ​​in their own language, while also reducing the burden of multilingual support for information providers. Throughout this entire system, convenient and efficient information sharing is achieved for both users and information providers.

[0070] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0071] Step 1:

[0072] The user's device captures screen information from applications and websites selected by the user. The input to this process is the state of the screen the user is viewing, and the output is the captured screenshot or text data. Specifically, the device uses a screen capture API to collect data and saves it in a formatted data format for further processing.

[0073] Step 2:

[0074] The terminal sends the acquired data to the server in a standard format. The input is the screenshot or text data obtained in step 1, and the output is the data packet sent to the server. For security purposes, the terminal encrypts and transmits the data using the HTTPS protocol.

[0075] Step 3:

[0076] The server receives data transmitted from the terminal and performs analysis using data processing tools. The input data consists of images and text information, and the output consists of analyzed text and visual information. Specific processing includes extracting characters from image data using optical character recognition technology.

[0077] Step 4:

[0078] The server translates the data analyzed using a generative AI model. The input is the text data obtained in step 3, and the output is the text translated into the specified language. A prompt is used to perform the translation; for example, entering "Translate this text into English" will perform the translation between multiple languages.

[0079] Step 5:

[0080] The server resends the translated data to the user's device. The input is the translated text, and the output is the display data sent to the device. The server formats the translation results appropriately and sends them to the device via the network protocol.

[0081] Step 6:

[0082] The terminal integrates the received translation data into the user interface and displays it to the user. The input is the translated data received in step 5, and the output is an information screen in the user's native language that the user can view. The terminal adjusts the font size and layout for user readability and presents information that is updated in real time.

[0083] (Application Example 1)

[0084] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0085] When visitors and travelers shop in foreign countries, language barriers can be a significant obstacle in understanding product information. In particular, accurate and rapid translation is essential for accurately understanding product labels and ingredient information. However, traditional methods require considerable effort to implement multilingual support, making it difficult for users. To address this challenge, an effective and efficient means of translating and displaying product information is necessary.

[0086] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0087] In this invention, the server includes means for a translation agent launched on the user's terminal to collect text data and visual data via a digital information acquisition means; computing device processing means for analyzing the collected text data and visual data and converting them into a translatable format; translation execution means for translating the analyzed data into multiple languages ​​using a generation AI; and means for the user to acquire product information via a camera, translate that information into their native language, and provide it to the customer. This enables visitors to instantly understand product information in their native language and make shopping smoother.

[0088] A "translation agent" is a software module that runs on the user's device and automatically collects and translates digital information.

[0089] "Means of acquiring digital information" refers to an interface or technology for acquiring text data and visual data through a terminal.

[0090] "Computational device processing means" refers to a computing device or process for analyzing collected data and converting it into a translatable format.

[0091] "Generative AI" refers to artificial intelligence models used to translate input data into multiple languages.

[0092] "Translation execution means" refers to the hardware or software process necessary to actually translate the analyzed data.

[0093] A "display interface" is a screen or application used to visually present translated information to the user.

[0094] "Visual data" is a general term for information that can be recognized visually, such as images and videos.

[0095] "Optical character recognition technology" is a technology for extracting character information from images.

[0096] "Product information" refers to detailed descriptions of the product, such as price and ingredients.

[0097] This invention constructs a system that provides product information to users in multiple languages ​​using a translation agent launched on the user's terminal. The terminal has a built-in camera, which is used to acquire product information in the store as visual data. The visual data is collected by a digital information acquisition means on the terminal, analyzed by a computing device processing means, and then converted into a translatable format. Subsequently, the analyzed data is translated using a generation AI and transmitted to the user's terminal and integrated into the display interface.

[0098] This process utilizes optical character recognition (OCR) technologies such as Google® Cloud Vision API to extract textual information from visual data and treat it as the target for translation. External translation services such as Google Translate API can be used for translation. In addition, generative AI models such as OpenAI® GPT are used to provide customers with natural and contextually relevant translation results.

[0099] As a concrete example, when a foreign tourist picks up an item in a souvenir shop, they use their smartphone camera to focus on the product label. At that moment, OCR technology recognizes the text information, instantly translates it into the desired language, and displays it on the smartphone screen. As an example of a prompt sentence in this process, the following sentence is provided to the generating AI model: "Translate the following Japanese product label text into English: 'Matcha (Uji Tea) 100g'. Provide a consumer-friendly description of the tea's taste and usage in a comprehensive manner." and the translation is optimized.

[0100] This invention allows users to instantly understand product information in their own language, enabling them to shop smoothly in foreign countries.

[0101] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0102] Step 1:

[0103] The user activates the device's camera and takes a picture of the product label. The captured image data is collected by a digital information acquisition system. The input is image data from the camera, and the device prepares this data to send to the next step.

[0104] Step 2:

[0105] The terminal sends the collected image data to the server. The server uses optical character recognition (OCR) technology to analyze the image data and extract text information. The input is image data, and the output is extracted text data. This text data is ready for translation.

[0106] Step 3:

[0107] The server translates text data extracted using generative AI. The server inputs prompt sentences into the generative AI model and translates text into multiple languages ​​based on instructions such as "Translate the following product label text into English: 'Matcha (Uji Tea) 100g'." The input is text data, and the output is translated text data.

[0108] Step 4:

[0109] The translated text data is sent from the server to the terminal. The terminal uses this data to visually display it to the user and integrate it into the interface. The input is the translated text data, and the output is the translated information displayed on the user's screen. This allows the user to view product information in their native language.

[0110] This series of processes allows users to easily understand and utilize product information in foreign countries.

[0111] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0112] This invention is a system that combines a translation agent with an emotion engine to enable the provision of multilingual information tailored to the user's emotions. The aim is to achieve communication that transcends language barriers and provide users with a more personal and enriching experience.

[0113] At system startup, the user activates the translation agent on their device. The device collects text and image data from specified applications and websites and sends it to the server. This data is converted into a translatable format and translated into the specified language by a generative AI. In particular, text information contained in image data is extracted using optical character recognition technology and targeted for translation.

[0114] The emotion engine, a key feature of this invention, analyzes user actions and inputs in real time to recognize the user's emotions. For example, if a user is urgently seeking information, the engine detects this urgency and presents the translated content concisely and intuitively. Conversely, in situations where the user is taking their time to review the information, the engine adjusts to provide a more detailed translation.

[0115] The translated data is sent from the server to the terminal and displayed on the interface using display control means, with a tone and expression that matches the user's emotions. In this way, users can receive translated content in their native language in a way that suits their emotions, enabling a more natural and satisfying information experience.

[0116] For example, when a tourist visiting Japan tries to make a restaurant reservation, if the emotional engine recognizes their impatience, it will quickly provide a simplified reservation method, resulting in a smooth reservation experience. Similarly, when a tourist is looking at a travel guide, if the emotional engine recognizes their heightened curiosity and interest, it will provide more detailed information, enriching the experience.

[0117] In this system, the combination of emotion recognition by an emotion engine and multilingual translation by generative AI enables flexible information provision tailored to the user's situation and emotions, playing a role in making information acquisition in Japan more convenient.

[0118] The following describes the processing flow.

[0119] Step 1:

[0120] The user launches the translation agent on their device and opens the desired application or website. The device collects text and image data from the app or website, and the emotion engine detects the user's current emotional state.

[0121] Step 2:

[0122] The device uses an emotion engine to analyze the user's operation speed and screen touch frequency, recognizing the user's emotions in real time. This information is sent to the server along with the data.

[0123] Step 3:

[0124] The server analyzes the received data and organizes the text data. Text information within images is extracted using optical character recognition technology. The server also analyzes the user's sentiment data to determine if appropriate adjustments to the translation are necessary.

[0125] Step 4:

[0126] The server translates the data analyzed using generative AI into the specified language. In doing so, it uses a tone and expression that reflects the recognized user's emotions. For example, it provides a concise translation if the user is in a hurry, and a more detailed translation if they are excited.

[0127] Step 5:

[0128] The server sends the translation results to the terminal. The terminal then presents the user with an adapted interface based on these translation results. The displayed information is provided in a format that harmonizes with the user's perceived emotions.

[0129] Step 6:

[0130] Users can review translations optimized to their emotions and obtain the information they need. This allows for a more natural and less stressful information retrieval experience.

[0131] (Example 2)

[0132] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0133] In today's globalized society, smooth communication between users who speak different languages ​​is essential. However, simply translating languages ​​makes it difficult to provide appropriate information that takes into account the user's feelings and circumstances. Therefore, a system is needed that can provide more personalized and context-appropriate information.

[0134] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0135] In this invention, the server includes an information processing device, an information conversion device, and an emotion analysis device. This makes it possible to provide a multilingual translation service that is adjusted according to the user's emotions and feelings.

[0136] A "translation agent" is an application that runs on the user's device and assists with translation between different languages.

[0137] "Electronic information acquisition means" refers to functions for collecting digital data through terminals, and is responsible for obtaining necessary information from applications and websites.

[0138] "Digital data" refers to electronically processed information such as text, images, and audio.

[0139] An "information processing device" is a device that has the function of analyzing collected digital data and converting it into a translatable format.

[0140] "Generative AI" refers to artificial intelligence technology that learns from large amounts of data and provides appropriate responses and processing results in response to input.

[0141] An "information conversion means" is a device or process that has the function of translating information analyzed using generative AI into multiple languages.

[0142] "Emotional analysis tools" are devices or algorithms that analyze the user's emotions and circumstances, and adjust the system's response and translation content based on that information.

[0143] "Display control device means" refers to a device or process for appropriately displaying translated data on the user's terminal.

[0144] This invention is a multilingual translation system that enables users to communicate smoothly between different languages ​​and provides information based on emotions.

[0145] The user launches a dedicated translation agent on their device. This agent retrieves digital data, including text and images, from sources such as applications and websites. Text information within images is extracted using optical character recognition (OCR) technology installed on the device. Standard OCR software can be used for this process.

[0146] Digital data collected by the terminal is sent to a server. The server analyzes the received data through an information processing device and converts it into a translatable format. Using a generative AI model, the data is translated into multiple languages. In this process, efficient translation is performed using GPT and other language processing AI.

[0147] The translated information is then appropriately adjusted according to the user's emotional state using sentiment analysis tools. This adjustment generates information with a tone and expression that matches the user's emotions. For example, if the user is anxious, the information will be presented concisely, while users seeking a more detailed understanding will be provided with more detailed information.

[0148] Ultimately, the translated data from the server is displayed appropriately on the terminal by the display control device. This allows users to receive information in their native language in an emotionally appropriate manner.

[0149] For example, when a tourist visiting Japan uses a translation agent to make a restaurant reservation, if the sentiment analysis system detects that the user is in a hurry, the process will be simplified. Also, if the sentiment analysis system determines that the user is seeking more information while viewing a tourist guide, additional detailed tourist information will be provided.

[0150] An example of a prompt message would be, "Please explain how the emotion engine works to optimize information delivery when tourists are researching information about Japanese tourist destinations." In this way, this system enables a personalized information experience that goes beyond the scope of language translation.

[0151] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0152] Step 1:

[0153] The user activates the translation agent on their device. This action initiates the entire system process. The input consists of the type and format of information to be translated, specified by the user. The output is that the agent is activated and ready to collect data. Specifically, the user interface opens, and the system awaits further instructions from the user.

[0154] Step 2:

[0155] The device collects digital data from user-specified information sources (e.g., websites, applications). Input is the URL or file path of the information source, and output is extracted as text or image data. In particular, for image data, OCR technology is used to extract text information. Through this specific operation, the device provides raw data in an easy-to-use digital format.

[0156] Step 3:

[0157] The terminal transmits the collected digital data to the server. The input is digital data stored within the terminal, which is converted into a format that the server can process. As output, the data transmitted via a secure protocol is stored on the server. Specifically, the SSL / TLS protocol is used to transmit the data while maintaining its confidentiality and integrity.

[0158] Step 4:

[0159] The server analyzes the received digital data and converts it into a translatable format. The input is the transmitted digital data, and the output is structured data, parsed individually for each language. Specifically, a language detection algorithm is applied, and the data is converted into an appropriate data model.

[0160] Step 5:

[0161] The server translates information analyzed using a generative AI model into multiple languages. The input is structured data, and the output is text translated into multiple target languages. Specifically, the AI ​​model performs natural translation that takes context and nuance into account.

[0162] Step 6:

[0163] The server performs sentiment analysis and adjusts the translated information according to the user's emotions. The input is the user's action history and reactions, and the output is translated data with optimized tone and expression. Specifically, it uses sentiment recognition software to select the most appropriate expression based on the user's situation.

[0164] Step 7:

[0165] The server sends the final translation result to the terminal. The input is the adjusted translation data, and the output is the information displayed on the terminal. Specifically, the data is sent back using the HTTP protocol and converted into a format suitable for display.

[0166] Step 8:

[0167] The terminal displays the received data through a user interface. The input is translated data sent from the server, and the output is information that the user interprets in a way that suits their emotions. Specifically, the translated content is presented on the GUI with the most appropriate font and layout.

[0168] (Application Example 2)

[0169] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0170] When users access information in different languages, there is a challenge in providing more personalized and emotionally relevant information across language barriers. Furthermore, adjusting the amount of information and the tone of translation to suit the user's emotional state has not been adequately done in the past. This limits the improvement of the user experience.

[0171] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0172] In this invention, the server includes means for collecting text data and image data, means for converting them into a translatable format, and means for integrating the translated data. This makes it possible to provide multilingual information tailored to the emotional state of the user.

[0173] A "translation agent" is a program that runs on the user's device and performs translation between multiple languages.

[0174] An "electronic information acquisition means" is an interface for acquiring electronic information such as text data and image data.

[0175] A "server processing means" is a computer system that processes collected data to analyze it and convert it into a translatable format.

[0176] A "translation execution method" is a program that has the function of translating data analyzed using a generative AI into multiple languages.

[0177] "Display control means" refers to a function that transmits translated data to the user's terminal and controls it so that it is properly integrated into the display interface.

[0178] "Emotional analysis means" refers to a function that analyzes the user's emotional state in real time and adjusts the translation content based on the results.

[0179] The system for carrying out this invention consists of a user terminal, a server, a translation agent, and sentiment analysis means. The server receives text and image data collected from the terminal and analyzes it. In doing so, optical character recognition technology is used to convert the collected data into a translatable format. The translation agent translates the data into multiple languages ​​using a generative AI model. The server also analyzes the user's emotional state in real time via the sentiment analysis means and adjusts the translation content based on the results. This adjustment is reflected in the level of detail and tone of the information.

[0180] Ultimately, the server sends the translated data to the terminal, and the display control system shows an integrated interface tailored to the user's emotions. This allows the user to use information in a more personalized and satisfying way.

[0181] A concrete example is a robotic guide at a tourist destination. This guide provides appropriate translations based on the content of the tourist's questions and their emotions. For example, when a tourist asks for a detailed explanation of a sculpture they are interested in, the system detects the tourist's excitement using emotion analysis and provides information accordingly in their native language.

[0182] An example of a prompt is, "Provide detailed information about this sculpture in the native language of interested visitors." This example demonstrates how a generative AI model can be used to provide information tailored to the user's emotions.

[0183] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0184] Step 1:

[0185] The user's device launches the translation agent. The device uses electronic data acquisition means to collect text and image data specified by the user. The input data includes information that the user is interested in. The output is raw text and image data that is sent to the server.

[0186] Step 2:

[0187] The server analyzes the received data. Here, optical character recognition technology is used to extract character information from the image data. The input is the text and image data collected in step 1. As a result of the data processing, text data in a translatable format is output.

[0188] Step 3:

[0189] The server uses a generated AI model to translate translatable text data into multiple specified languages. The input is the translatable text data obtained in step 2. As a result of the translation, multilingual text data is output.

[0190] Step 4:

[0191] The server uses sentiment analysis tools to analyze the user's emotional state based on the information and actions the user has entered. Inputs include data such as user action logs and audio feedback. Outputs are metadata indicating the user's emotional state.

[0192] Step 5:

[0193] The server adjusts the translated multilingual data based on the emotional state. Specifically, it provides a concise translation to users in a hurry and a detailed translation to users who are showing interest. The input is the translated data from step 3 and the emotional metadata from step 4. The output is the adjusted multilingual data.

[0194] Step 6:

[0195] The server sends the adjusted translation data to the user's terminal and integrates it appropriately on the interface using display control means. The terminal displays information that matches the user's emotions. The input is the multilingual data adjusted in step 5, and the output is the display interface on the user's terminal.

[0196] Step 7:

[0197] The user receives the displayed information and requests additional information or provides feedback as needed. This serves as foundational data for providing further information and making adjustments. The input in this step is the displayed information, and the output is new input and feedback from the user.

[0198] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0199] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0200] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0201] [Second Embodiment]

[0202] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0203] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0204] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0205] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0206] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0207] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0208] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0209] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0210] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0211] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0212] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0213] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0214] The system of this invention was developed to improve user convenience and reduce the cost and effort of multilingual support. In this system, a translation agent runs on the user's terminal and collects and translates information from specified applications and websites to provide the user with information in their native language.

[0215] The terminal acquires screen data from applications and websites accessed by the user and sends it to the server in a specific format. The server analyzes the received data and extracts text and image data according to the format. This extracted data is translated into multiple languages ​​by a generative AI. In particular, for image data, optical character recognition technology is used as needed to extract character information and include it in the translation.

[0216] The translated data is sent from the server to the terminal and integrated into the interface in a user-friendly format by display control means. This entire process allows users to easily understand information in their native language and conduct their activities smoothly within Japan.

[0217] For example, when tourists visiting Japan use a local train transfer guidance app, they can simply open the app, and a translation agent will automatically activate, allowing them to view transfer information in their native language. This enables them to travel comfortably without feeling a language barrier, making their sightseeing even more enjoyable.

[0218] This invention is expected to not only expedite the provision of information to tourists but also significantly reduce the burden of multilingual support on public transportation and service providers within Japan. In this way, the present invention will improve the efficiency of multilingual support and contribute to building a win-win relationship between foreign visitors to Japan and Japanese tourism businesses.

[0219] The following describes the processing flow.

[0220] Step 1:

[0221] The user launches the translation agent on their device and opens the application or website to be translated. The device automatically captures the content on the screen and extracts the text and image data that needs to be translated.

[0222] Step 2:

[0223] The terminal sends the extracted text and image data to the server as an analysis request. This request also includes the user's desired target language for translation.

[0224] Step 3:

[0225] The server analyzes the received data, organizing the text and extracting character information from images. Using optical character recognition (OCR) technology, it identifies characters within images and converts them into a translatable format.

[0226] Step 4:

[0227] The server uses generative AI to translate the analyzed text and image data into the specified language. During translation, adjustments are made to generate the most appropriate expression based on the context.

[0228] Step 5:

[0229] The server sends the translated data to the terminal. The terminal then uses this translation to reconfigure the application or website interface in its native language.

[0230] Step 6:

[0231] Users can view the translation results displayed on their device and obtain the necessary information. This allows them to use apps and websites without feeling any language barriers.

[0232] (Example 1)

[0233] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0234] In today's world, where multilingualism is essential, understanding information provided in different languages ​​poses a significant obstacle for users of information processing devices who are not proficient in those languages. Furthermore, information providers face the challenge of incurring substantial costs for multilingual support.

[0235] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0236] In this invention, the server includes means for a language conversion agent activated on the user's information processing device to collect textual and visual information via a target content acquisition means, data processing means for analyzing the collected textual and visual information and converting it into a language-convertible format, and conversion execution means for converting the analyzed data into multiple languages ​​using a generation AI. This allows users to easily obtain information in their native language and reduces the costs associated with multilingual support.

[0237] An "information processing device" is an electronic device that allows users to acquire and process information via the internet or applications.

[0238] A "language conversion agent" is a software program that operates on an information processing device and collects textual and visual information from content accessed by the user.

[0239] "Content acquisition means" refers to the process or technology used by an information processing device to acquire necessary data from applications or websites it accesses.

[0240] "Textual information" refers to a form of information recorded as text data, such as strings of characters found in documents and emails.

[0241] "Visual information" refers to data provided in a visual format, including elements found in images and videos.

[0242] "Data processing means" refers to processes and equipment that analyze collected textual and visual information and convert it for use in other formats or applications.

[0243] "Generative AI" is a system that uses artificial intelligence technology to generate new information or output from input data according to a specified task.

[0244] "Optical character recognition technology" is a technology that converts characters contained in visual information into electronic text data.

[0245] The system of this invention takes the form of a language conversion agent operating within an information processing device in order to improve user convenience. The information processing device acquires content from applications and websites accessed by the user and converts it into a standardized data format. In this acquisition process, text and image data are collected using screen capture technology and APIs.

[0246] The server receives data acquired from the terminal and performs analysis using its built-in data processing capabilities. During the analysis, textual information is distinguished from visual information, and the visual information is converted into text using optical character recognition technology (e.g., open-source OCR systems). This process needs to be particularly fast and accurate in applications requiring multilingual support.

[0247] The analyzed data is then translated into the specified language by the generating AI. The generating AI performs multilingual translation based on the prompt message specified during the language conversion. An example of such a prompt message is, "Translate this text into the target language."

[0248] The translated data is sent from the server to the user's information processing device, and the terminal integrates that data into the user interface using a display management mechanism. For example, when a user sightseeing in Japan uses a local guide app, the terminal automatically displays the translated content, allowing them to check information in real time without experiencing language barriers.

[0249] This allows users to easily understand information provided in different languages ​​in their own language, while also reducing the burden of multilingual support for information providers. Throughout this entire system, convenient and efficient information sharing is achieved for both users and information providers.

[0250] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0251] Step 1:

[0252] The user's device captures screen information from applications and websites selected by the user. The input to this process is the state of the screen the user is viewing, and the output is the captured screenshot or text data. Specifically, the device uses a screen capture API to collect data and saves it in a formatted data format for further processing.

[0253] Step 2:

[0254] The terminal sends the acquired data to the server in a standard format. The input is the screenshot or text data obtained in step 1, and the output is the data packet sent to the server. For security purposes, the terminal encrypts and transmits the data using the HTTPS protocol.

[0255] Step 3:

[0256] The server receives data transmitted from the terminal and performs analysis using data processing tools. The input data consists of images and text information, and the output consists of analyzed text and visual information. Specific processing includes extracting characters from image data using optical character recognition technology.

[0257] Step 4:

[0258] The server translates the data analyzed using a generative AI model. The input is the text data obtained in step 3, and the output is the text translated into the specified language. A prompt is used to perform the translation; for example, entering "Translate this text into English" will perform the translation between multiple languages.

[0259] Step 5:

[0260] The server resends the translated data to the user's device. The input is the translated text, and the output is the display data sent to the device. The server formats the translation results appropriately and sends them to the device via the network protocol.

[0261] Step 6:

[0262] The terminal integrates the received translation data into the user interface and displays it to the user. The input is the translated data received in step 5, and the output is an information screen in the user's native language that the user can view. The terminal adjusts the font size and layout for user readability and presents information that is updated in real time.

[0263] (Application Example 1)

[0264] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0265] When visitors and travelers shop in foreign countries, language barriers can be a significant obstacle in understanding product information. In particular, accurate and rapid translation is essential for accurately understanding product labels and ingredient information. However, traditional methods require considerable effort to implement multilingual support, making it difficult for users. To address this challenge, an effective and efficient means of translating and displaying product information is necessary.

[0266] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0267] In this invention, the server includes means for a translation agent launched on the user's terminal to collect text data and visual data via a digital information acquisition means; computing device processing means for analyzing the collected text data and visual data and converting them into a translatable format; translation execution means for translating the analyzed data into multiple languages ​​using a generation AI; and means for the user to acquire product information via a camera, translate that information into their native language, and provide it to the customer. This enables visitors to instantly understand product information in their native language and make shopping smoother.

[0268] A "translation agent" is a software module that runs on the user's device and automatically collects and translates digital information.

[0269] "Means of acquiring digital information" refers to an interface or technology for acquiring text data and visual data through a terminal.

[0270] "Computational device processing means" refers to a computing device or process for analyzing collected data and converting it into a translatable format.

[0271] "Generative AI" refers to artificial intelligence models used to translate input data into multiple languages.

[0272] "Translation execution means" refers to the hardware or software process necessary to actually translate the analyzed data.

[0273] A "display interface" is a screen or application used to visually present translated information to the user.

[0274] "Visual data" is a general term for information that can be recognized visually, such as images and videos.

[0275] "Optical character recognition technology" is a technology for extracting character information from images.

[0276] "Product information" refers to detailed descriptions of the product, such as price and ingredients.

[0277] This invention constructs a system that provides product information to users in multiple languages ​​using a translation agent launched on the user's terminal. The terminal has a built-in camera, which is used to acquire product information in the store as visual data. The visual data is collected by a digital information acquisition means on the terminal, analyzed by a computing device processing means, and then converted into a translatable format. Subsequently, the analyzed data is translated using a generation AI and transmitted to the user's terminal and integrated into the display interface.

[0278] This process utilizes optical character recognition (OCR) technologies such as the Google Cloud Vision API to extract textual information from visual data and treat it as the target for translation. External translation services such as the Google Translate API can be used for translation. Furthermore, AI models such as OpenAI GPT are used as the generative AI model to provide customers with natural and contextually relevant translation results.

[0279] As a specific example, when a foreign tourist picks up a product in a souvenir shop, they use the camera of their smartphone to focus on the product label. At that moment, the character information is recognized by OCR technology and immediately translated into the desired language and displayed on the screen of the smartphone. As an example of the prompt text in this process, the sentence "Translate the following Japanese product label text into English: 'Matcha (Uji tea) 100g'. Provide a consumer-friendly description of the tea's taste and usage in a comprehensive manner." is provided to the generative AI model to optimize the translation.

[0280] According to this invention, users can immediately understand product information in their native language, enabling smooth shopping in foreign countries.

[0281] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0282] Step 1:

[0283] The user activates the camera of the terminal to take a picture of the product label. The captured image data is collected by the digital information acquisition means. The input is the image data through the camera, and the terminal prepares to send this data to the next step.

[0284] Step 2:

[0285] The terminal sends the collected image data to the server. The server analyzes the image data using optical character recognition technology (OCR) and extracts the character information. The input is the image data, and the output is the extracted text data. This text data is in a state ready for translation.

[0286] Step 3:

[0287] The server translates text data extracted using generative AI. The server inputs prompt sentences into the generative AI model and translates text into multiple languages ​​based on instructions such as "Translate the following product label text into English: 'Matcha (Uji Tea) 100g'." The input is text data, and the output is translated text data.

[0288] Step 4:

[0289] The translated text data is sent from the server to the terminal. The terminal uses this data to visually display it to the user and integrate it into the interface. The input is the translated text data, and the output is the translated information displayed on the user's screen. This allows the user to view product information in their native language.

[0290] This series of processes allows users to easily understand and utilize product information in foreign countries.

[0291] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0292] This invention is a system that combines a translation agent with an emotion engine to enable the provision of multilingual information tailored to the user's emotions. The aim is to achieve communication that transcends language barriers and provide users with a more personal and enriching experience.

[0293] At system startup, the user activates the translation agent on their device. The device collects text and image data from specified applications and websites and sends it to the server. This data is converted into a translatable format and translated into the specified language by a generative AI. In particular, text information contained in image data is extracted using optical character recognition technology and targeted for translation.

[0294] The emotion engine, a key feature of this invention, analyzes user actions and inputs in real time to recognize the user's emotions. For example, if a user is urgently seeking information, the engine detects this urgency and presents the translated content concisely and intuitively. Conversely, in situations where the user is taking their time to review the information, the engine adjusts to provide a more detailed translation.

[0295] The translated data is sent from the server to the terminal and displayed on the interface using display control means, with a tone and expression that matches the user's emotions. In this way, users can receive translated content in their native language in a way that suits their emotions, enabling a more natural and satisfying information experience.

[0296] For example, when a tourist visiting Japan tries to make a restaurant reservation, if the emotional engine recognizes their impatience, it will quickly provide a simplified reservation method, resulting in a smooth reservation experience. Similarly, when a tourist is looking at a travel guide, if the emotional engine recognizes their heightened curiosity and interest, it will provide more detailed information, enriching the experience.

[0297] In this system, the combination of emotion recognition by an emotion engine and multilingual translation by generative AI enables flexible information provision tailored to the user's situation and emotions, playing a role in making information acquisition in Japan more convenient.

[0298] The following describes the processing flow.

[0299] Step 1:

[0300] The user launches the translation agent on their device and opens the desired application or website. The device collects text and image data from the app or website, and the emotion engine detects the user's current emotional state.

[0301] Step 2:

[0302] The terminal uses an emotion engine to analyze the user's operation speed, touch frequency on the screen, etc., and recognizes the user's emotions in real time. This information is sent to the server together with the data.

[0303] Step 3:

[0304] The server analyzes the received data and organizes the text data. The character information in the image is extracted by optical character recognition technology. The server also analyzes the user's emotion data and determines whether it is necessary to adjust the appropriate expression in the translation.

[0305] Step 4:

[0306] The server translates the data analyzed using the generative AI into the specified language. At this time, tones and expressions that reflect the recognized emotions of the user are used. For example, a concise translation is provided when the user is in a hurry, and a detailed translation is provided when the user is excited.

[0307] Step 5:

[0308] The server sends the translation result to the terminal. The terminal presents the adapted interface to the user based on this translation result. The information displayed is provided in a form that harmonizes with the recognized emotions of the user.

[0309] Step 6:

[0310] The user checks the translation result optimized according to the emotion and obtains the necessary information. By doing so, the user can obtain information in a more natural and less stressful way.

[0311] (Example 2)

[0312] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0313] In today's globalized society, smooth communication between users who speak different languages ​​is essential. However, simply translating languages ​​makes it difficult to provide appropriate information that takes into account the user's feelings and circumstances. Therefore, a system is needed that can provide more personalized and context-appropriate information.

[0314] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0315] In this invention, the server includes an information processing device, an information conversion device, and an emotion analysis device. This makes it possible to provide a multilingual translation service that is adjusted according to the user's emotions and feelings.

[0316] A "translation agent" is an application that runs on the user's device and assists with translation between different languages.

[0317] "Electronic information acquisition means" refers to functions for collecting digital data through terminals, and is responsible for obtaining necessary information from applications and websites.

[0318] "Digital data" refers to electronically processed information such as text, images, and audio.

[0319] An "information processing device" is a device that has the function of analyzing collected digital data and converting it into a translatable format.

[0320] "Generative AI" refers to artificial intelligence technology that learns from large amounts of data and provides appropriate responses and processing results in response to input.

[0321] An "information conversion means" is a device or process that has the function of translating information analyzed using generative AI into multiple languages.

[0322] "Emotional analysis tools" are devices or algorithms that analyze the user's emotions and circumstances, and adjust the system's response and translation content based on that information.

[0323] "Display control device means" refers to a device or process for appropriately displaying translated data on the user's terminal.

[0324] This invention is a multilingual translation system that enables users to communicate smoothly between different languages ​​and provides information based on emotions.

[0325] The user launches a dedicated translation agent on their device. This agent retrieves digital data, including text and images, from sources such as applications and websites. Text information within images is extracted using optical character recognition (OCR) technology installed on the device. Standard OCR software can be used for this process.

[0326] Digital data collected by the terminal is sent to a server. The server analyzes the received data through an information processing device and converts it into a translatable format. Using a generative AI model, the data is translated into multiple languages. In this process, efficient translation is performed using GPT and other language processing AI.

[0327] The translated information is then appropriately adjusted according to the user's emotional state using sentiment analysis tools. This adjustment generates information with a tone and expression that matches the user's emotions. For example, if the user is anxious, the information will be presented concisely, while users seeking a more detailed understanding will be provided with more detailed information.

[0328] Ultimately, the translated data from the server is displayed appropriately on the terminal by the display control device. This allows users to receive information in their native language in an emotionally appropriate manner.

[0329] For example, when a tourist visiting Japan uses a translation agent to make a restaurant reservation, if the sentiment analysis system detects that the user is in a hurry, the process will be simplified. Also, if the sentiment analysis system determines that the user is seeking more information while viewing a tourist guide, additional detailed tourist information will be provided.

[0330] An example of a prompt message would be, "Please explain how the emotion engine works to optimize information delivery when tourists are researching information about Japanese tourist destinations." In this way, this system enables a personalized information experience that goes beyond the scope of language translation.

[0331] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0332] Step 1:

[0333] The user activates the translation agent on their device. This action initiates the entire system process. The input consists of the type and format of information to be translated, specified by the user. The output is that the agent is activated and ready to collect data. Specifically, the user interface opens, and the system awaits further instructions from the user.

[0334] Step 2:

[0335] The device collects digital data from user-specified information sources (e.g., websites, applications). Input is the URL or file path of the information source, and output is extracted as text or image data. In particular, for image data, OCR technology is used to extract text information. Through this specific operation, the device provides raw data in an easy-to-use digital format.

[0336] Step 3:

[0337] The terminal transmits the collected digital data to the server. The input is digital data stored within the terminal, which is converted into a format that the server can process. As output, the data transmitted via a secure protocol is stored on the server. Specifically, the SSL / TLS protocol is used to transmit the data while maintaining its confidentiality and integrity.

[0338] Step 4:

[0339] The server analyzes the received digital data and converts it into a translatable format. The input is the transmitted digital data, and the output is structured data, parsed individually for each language. Specifically, a language detection algorithm is applied, and the data is converted into an appropriate data model.

[0340] Step 5:

[0341] The server translates information analyzed using a generative AI model into multiple languages. The input is structured data, and the output is text translated into multiple target languages. Specifically, the AI ​​model performs natural translation that takes context and nuance into account.

[0342] Step 6:

[0343] The server performs sentiment analysis and adjusts the translated information according to the user's emotions. The input is the user's action history and reactions, and the output is translated data with optimized tone and expression. Specifically, it uses sentiment recognition software to select the most appropriate expression based on the user's situation.

[0344] Step 7:

[0345] The server sends the final translation result to the terminal. The input is the adjusted translation data, and the output is the information displayed on the terminal. Specifically, the data is sent back using the HTTP protocol and converted into a format suitable for display.

[0346] Step 8:

[0347] The terminal displays the received data through a user interface. The input is translated data sent from the server, and the output is information that the user interprets in a way that suits their emotions. Specifically, the translated content is presented on the GUI with the most appropriate font and layout.

[0348] (Application Example 2)

[0349] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0350] When users access information in different languages, there is a challenge in providing more personalized and emotionally relevant information across language barriers. Furthermore, adjusting the amount of information and the tone of translation to suit the user's emotional state has not been adequately done in the past. This limits the improvement of the user experience.

[0351] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0352] In this invention, the server includes means for collecting text data and image data, means for converting them into a translatable format, and means for integrating the translated data. This makes it possible to provide multilingual information tailored to the emotional state of the user.

[0353] A "translation agent" is a program that runs on the user's device and performs translation between multiple languages.

[0354] An "electronic information acquisition means" is an interface for acquiring electronic information such as text data and image data.

[0355] A "server processing means" is a computer system that processes collected data to analyze it and convert it into a translatable format.

[0356] A "translation execution method" is a program that has the function of translating data analyzed using a generative AI into multiple languages.

[0357] "Display control means" refers to a function that transmits translated data to the user's terminal and controls it so that it is properly integrated into the display interface.

[0358] "Emotional analysis means" refers to a function that analyzes the user's emotional state in real time and adjusts the translation content based on the results.

[0359] The system for carrying out this invention consists of a user terminal, a server, a translation agent, and sentiment analysis means. The server receives text and image data collected from the terminal and analyzes it. In doing so, optical character recognition technology is used to convert the collected data into a translatable format. The translation agent translates the data into multiple languages ​​using a generative AI model. The server also analyzes the user's emotional state in real time via the sentiment analysis means and adjusts the translation content based on the results. This adjustment is reflected in the level of detail and tone of the information.

[0360] Ultimately, the server sends the translated data to the terminal, and the display control system shows an integrated interface tailored to the user's emotions. This allows the user to use information in a more personalized and satisfying way.

[0361] A concrete example is a robotic guide at a tourist destination. This guide provides appropriate translations based on the content of the tourist's questions and their emotions. For example, when a tourist asks for a detailed explanation of a sculpture they are interested in, the system detects the tourist's excitement using emotion analysis and provides information accordingly in their native language.

[0362] An example of a prompt is, "Provide detailed information about this sculpture in the native language of interested visitors." This example demonstrates how a generative AI model can be used to provide information tailored to the user's emotions.

[0363] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0364] Step 1:

[0365] The user's device launches the translation agent. The device uses electronic data acquisition means to collect text and image data specified by the user. The input data includes information that the user is interested in. The output is raw text and image data that is sent to the server.

[0366] Step 2:

[0367] The server analyzes the received data. Here, optical character recognition technology is used to extract character information from the image data. The input is the text and image data collected in step 1. As a result of the data processing, text data in a translatable format is output.

[0368] Step 3:

[0369] The server uses a generated AI model to translate translatable text data into multiple specified languages. The input is the translatable text data obtained in step 2. As a result of the translation, multilingual text data is output.

[0370] Step 4:

[0371] The server uses sentiment analysis tools to analyze the user's emotional state based on the information and actions the user has entered. Inputs include data such as user action logs and audio feedback. Outputs are metadata indicating the user's emotional state.

[0372] Step 5:

[0373] The server adjusts the translated multilingual data based on the emotional state. Specifically, it provides a concise translation to users in a hurry and a detailed translation to users who are showing interest. The input is the translated data from step 3 and the emotional metadata from step 4. The output is the adjusted multilingual data.

[0374] Step 6:

[0375] The server sends the adjusted translation data to the user's terminal and integrates it appropriately on the interface using display control means. The terminal displays information that matches the user's emotions. The input is the multilingual data adjusted in step 5, and the output is the display interface on the user's terminal.

[0376] Step 7:

[0377] The user receives the displayed information and requests additional information or provides feedback as needed. This serves as foundational data for providing further information and making adjustments. The input in this step is the displayed information, and the output is new input and feedback from the user.

[0378] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0379] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0380] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0381] [Third Embodiment]

[0382] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0383] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0384] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0385] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0386] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0387] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0388] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0389] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0390] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0391] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0392] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0393] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0394] The system of this invention was developed to improve user convenience and reduce the cost and effort of multilingual support. In this system, a translation agent runs on the user's terminal and collects and translates information from specified applications and websites to provide the user with information in their native language.

[0395] The terminal acquires screen data from applications and websites accessed by the user and sends it to the server in a specific format. The server analyzes the received data and extracts text and image data according to the format. This extracted data is translated into multiple languages ​​by a generative AI. In particular, for image data, optical character recognition technology is used as needed to extract character information and include it in the translation.

[0396] The translated data is sent from the server to the terminal and integrated into the interface in a user-friendly format by display control means. This entire process allows users to easily understand information in their native language and conduct their activities smoothly within Japan.

[0397] For example, when tourists visiting Japan use a local train transfer guidance app, they can simply open the app, and a translation agent will automatically activate, allowing them to view transfer information in their native language. This enables them to travel comfortably without feeling a language barrier, making their sightseeing even more enjoyable.

[0398] This invention is expected to not only expedite the provision of information to tourists but also significantly reduce the burden of multilingual support on public transportation and service providers within Japan. In this way, the present invention will improve the efficiency of multilingual support and contribute to building a win-win relationship between foreign visitors to Japan and Japanese tourism businesses.

[0399] The following describes the processing flow.

[0400] Step 1:

[0401] The user launches the translation agent on their device and opens the application or website to be translated. The device automatically captures the content on the screen and extracts the text and image data that needs to be translated.

[0402] Step 2:

[0403] The terminal sends the extracted text and image data to the server as an analysis request. This request also includes the user's desired target language for translation.

[0404] Step 3:

[0405] The server analyzes the received data, organizing the text and extracting character information from images. Using optical character recognition (OCR) technology, it identifies characters within images and converts them into a translatable format.

[0406] Step 4:

[0407] The server uses generative AI to translate the analyzed text and image data into the specified language. During translation, adjustments are made to generate the most appropriate expression based on the context.

[0408] Step 5:

[0409] The server sends the translated data to the terminal. The terminal then uses this translation to reconfigure the application or website interface in its native language.

[0410] Step 6:

[0411] Users can view the translation results displayed on their device and obtain the necessary information. This allows them to use apps and websites without feeling any language barriers.

[0412] (Example 1)

[0413] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0414] In today's world, where multilingualism is essential, understanding information provided in different languages ​​poses a significant obstacle for users of information processing devices who are not proficient in those languages. Furthermore, information providers face the challenge of incurring substantial costs for multilingual support.

[0415] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0416] In this invention, the server includes means for a language conversion agent activated on the user's information processing device to collect textual and visual information via a target content acquisition means, data processing means for analyzing the collected textual and visual information and converting it into a language-convertible format, and conversion execution means for converting the analyzed data into multiple languages ​​using a generation AI. This allows users to easily obtain information in their native language and reduces the costs associated with multilingual support.

[0417] An "information processing device" is an electronic device that allows users to acquire and process information via the internet or applications.

[0418] A "language conversion agent" is a software program that operates on an information processing device and collects textual and visual information from content accessed by the user.

[0419] "Content acquisition means" refers to the process or technology used by an information processing device to acquire necessary data from applications or websites it accesses.

[0420] "Textual information" refers to a form of information recorded as text data, such as strings of characters found in documents and emails.

[0421] "Visual information" refers to data provided in a visual format, including elements found in images and videos.

[0422] "Data processing means" refers to processes and equipment that analyze collected textual and visual information and convert it for use in other formats or applications.

[0423] "Generative AI" is a system that uses artificial intelligence technology to generate new information or output from input data according to a specified task.

[0424] "Optical character recognition technology" is a technology that converts characters contained in visual information into electronic text data.

[0425] The system of this invention takes the form of a language conversion agent operating within an information processing device in order to improve user convenience. The information processing device acquires content from applications and websites accessed by the user and converts it into a standardized data format. In this acquisition process, text and image data are collected using screen capture technology and APIs.

[0426] The server receives data acquired from the terminal and performs analysis using its built-in data processing capabilities. During the analysis, textual information is distinguished from visual information, and the visual information is converted into text using optical character recognition technology (e.g., open-source OCR systems). This process needs to be particularly fast and accurate in applications requiring multilingual support.

[0427] The analyzed data is then translated into the specified language by the generating AI. The generating AI performs multilingual translation based on the prompt message specified during the language conversion. An example of such a prompt message is, "Translate this text into the target language."

[0428] The translated data is sent from the server to the user's information processing device, and the terminal integrates that data into the user interface using a display management mechanism. For example, when a user sightseeing in Japan uses a local guide app, the terminal automatically displays the translated content, allowing them to check information in real time without experiencing language barriers.

[0429] This allows users to easily understand information provided in different languages ​​in their own language, while also reducing the burden of multilingual support for information providers. Throughout this entire system, convenient and efficient information sharing is achieved for both users and information providers.

[0430] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0431] Step 1:

[0432] The user's device captures screen information from applications and websites selected by the user. The input to this process is the state of the screen the user is viewing, and the output is the captured screenshot or text data. Specifically, the device uses a screen capture API to collect data and saves it in a formatted data format for further processing.

[0433] Step 2:

[0434] The terminal sends the acquired data to the server in a standard format. The input is the screenshot or text data obtained in step 1, and the output is the data packet sent to the server. For security purposes, the terminal encrypts and transmits the data using the HTTPS protocol.

[0435] Step 3:

[0436] The server receives data transmitted from the terminal and performs analysis using data processing tools. The input data consists of images and text information, and the output consists of analyzed text and visual information. Specific processing includes extracting characters from image data using optical character recognition technology.

[0437] Step 4:

[0438] The server translates the data analyzed using a generative AI model. The input is the text data obtained in step 3, and the output is the text translated into the specified language. A prompt is used to perform the translation; for example, entering "Translate this text into English" will perform the translation between multiple languages.

[0439] Step 5:

[0440] The server resends the translated data to the user's device. The input is the translated text, and the output is the display data sent to the device. The server formats the translation results appropriately and sends them to the device via the network protocol.

[0441] Step 6:

[0442] The terminal integrates the received translation data into the user interface and displays it to the user. The input is the translated data received in step 5, and the output is an information screen in the user's native language that the user can view. The terminal adjusts the font size and layout for user readability and presents information that is updated in real time.

[0443] (Application Example 1)

[0444] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0445] When visitors and travelers shop in foreign countries, language barriers can be a significant obstacle in understanding product information. In particular, accurate and rapid translation is essential for accurately understanding product labels and ingredient information. However, traditional methods require considerable effort to implement multilingual support, making it difficult for users. To address this challenge, an effective and efficient means of translating and displaying product information is necessary.

[0446] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0447] In this invention, the server includes means for a translation agent launched on the user's terminal to collect text data and visual data via a digital information acquisition means; computing device processing means for analyzing the collected text data and visual data and converting them into a translatable format; translation execution means for translating the analyzed data into multiple languages ​​using a generation AI; and means for the user to acquire product information via a camera, translate that information into their native language, and provide it to the customer. This enables visitors to instantly understand product information in their native language and make shopping smoother.

[0448] A "translation agent" is a software module that runs on the user's device and automatically collects and translates digital information.

[0449] "Means of acquiring digital information" refers to an interface or technology for acquiring text data and visual data through a terminal.

[0450] "Computational device processing means" refers to a computing device or process for analyzing collected data and converting it into a translatable format.

[0451] "Generative AI" refers to artificial intelligence models used to translate input data into multiple languages.

[0452] "Translation execution means" refers to the hardware or software process necessary to actually translate the analyzed data.

[0453] A "display interface" is a screen or application used to visually present translated information to the user.

[0454] "Visual data" is a general term for information that can be recognized visually, such as images and videos.

[0455] "Optical character recognition technology" is a technology for extracting character information from images.

[0456] "Product information" refers to detailed descriptions of the product, such as price and ingredients.

[0457] This invention constructs a system that provides product information to users in multiple languages ​​using a translation agent launched on the user's terminal. The terminal has a built-in camera, which is used to acquire product information in the store as visual data. The visual data is collected by a digital information acquisition means on the terminal, analyzed by a computing device processing means, and then converted into a translatable format. Subsequently, the analyzed data is translated using a generation AI and transmitted to the user's terminal and integrated into the display interface.

[0458] This process utilizes optical character recognition (OCR) technologies such as the Google Cloud Vision API to extract textual information from visual data and treat it as the target for translation. External translation services such as the Google Translate API can be used for translation. Furthermore, AI models such as OpenAI GPT are used as the generative AI model to provide customers with natural and contextually relevant translation results.

[0459] As a concrete example, when a foreign tourist picks up an item in a souvenir shop, they use their smartphone camera to focus on the product label. At that moment, OCR technology recognizes the text information, instantly translates it into the desired language, and displays it on the smartphone screen. As an example of a prompt sentence in this process, the following sentence is provided to the generating AI model: "Translate the following Japanese product label text into English: 'Matcha (Uji Tea) 100g'. Provide a consumer-friendly description of the tea's taste and usage in a comprehensive manner." and the translation is optimized.

[0460] This invention allows users to instantly understand product information in their own language, enabling them to shop smoothly in foreign countries.

[0461] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0462] Step 1:

[0463] The user activates the device's camera and takes a picture of the product label. The captured image data is collected by a digital information acquisition system. The input is image data from the camera, and the device prepares this data to send to the next step.

[0464] Step 2:

[0465] The terminal sends the collected image data to the server. The server uses optical character recognition (OCR) technology to analyze the image data and extract text information. The input is image data, and the output is extracted text data. This text data is ready for translation.

[0466] Step 3:

[0467] The server translates text data extracted using generative AI. The server inputs prompt sentences into the generative AI model and translates text into multiple languages ​​based on instructions such as "Translate the following product label text into English: 'Matcha (Uji Tea) 100g'." The input is text data, and the output is translated text data.

[0468] Step 4:

[0469] The translated text data is sent from the server to the terminal. The terminal uses this data to visually display it to the user and integrate it into the interface. The input is the translated text data, and the output is the translated information displayed on the user's screen. This allows the user to view product information in their native language.

[0470] This series of processes allows users to easily understand and utilize product information in foreign countries.

[0471] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0472] This invention is a system that combines a translation agent with an emotion engine to enable the provision of multilingual information tailored to the user's emotions. The aim is to achieve communication that transcends language barriers and provide users with a more personal and enriching experience.

[0473] At system startup, the user activates the translation agent on their device. The device collects text and image data from specified applications and websites and sends it to the server. This data is converted into a translatable format and translated into the specified language by a generative AI. In particular, text information contained in image data is extracted using optical character recognition technology and targeted for translation.

[0474] The emotion engine, a key feature of this invention, analyzes user actions and inputs in real time to recognize the user's emotions. For example, if a user is urgently seeking information, the engine detects this urgency and presents the translated content concisely and intuitively. Conversely, in situations where the user is taking their time to review the information, the engine adjusts to provide a more detailed translation.

[0475] The translated data is sent from the server to the terminal and displayed on the interface using display control means, with a tone and expression that matches the user's emotions. In this way, users can receive translated content in their native language in a way that suits their emotions, enabling a more natural and satisfying information experience.

[0476] For example, when a tourist visiting Japan tries to make a restaurant reservation, if the emotional engine recognizes their impatience, it will quickly provide a simplified reservation method, resulting in a smooth reservation experience. Similarly, when a tourist is looking at a travel guide, if the emotional engine recognizes their heightened curiosity and interest, it will provide more detailed information, enriching the experience.

[0477] In this system, the combination of emotion recognition by an emotion engine and multilingual translation by generative AI enables flexible information provision tailored to the user's situation and emotions, playing a role in making information acquisition in Japan more convenient.

[0478] The following describes the processing flow.

[0479] Step 1:

[0480] The user launches the translation agent on their device and opens the desired application or website. The device collects text and image data from the app or website, and the emotion engine detects the user's current emotional state.

[0481] Step 2:

[0482] The device uses an emotion engine to analyze the user's operation speed and screen touch frequency, recognizing the user's emotions in real time. This information is sent to the server along with the data.

[0483] Step 3:

[0484] The server analyzes the received data and organizes the text data. Text information within images is extracted using optical character recognition technology. The server also analyzes the user's sentiment data to determine if appropriate adjustments to the translation are necessary.

[0485] Step 4:

[0486] The server translates the data analyzed using generative AI into the specified language. In doing so, it uses a tone and expression that reflects the recognized user's emotions. For example, it provides a concise translation if the user is in a hurry, and a more detailed translation if they are excited.

[0487] Step 5:

[0488] The server sends the translation results to the terminal. The terminal then presents the user with an adapted interface based on these translation results. The displayed information is provided in a format that harmonizes with the user's perceived emotions.

[0489] Step 6:

[0490] Users can review translations optimized to their emotions and obtain the information they need. This allows for a more natural and less stressful information retrieval experience.

[0491] (Example 2)

[0492] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0493] In today's globalized society, smooth communication between users who speak different languages ​​is essential. However, simply translating languages ​​makes it difficult to provide appropriate information that takes into account the user's feelings and circumstances. Therefore, a system is needed that can provide more personalized and context-appropriate information.

[0494] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0495] In this invention, the server includes an information processing device, an information conversion device, and an emotion analysis device. This makes it possible to provide a multilingual translation service that is adjusted according to the user's emotions and feelings.

[0496] A "translation agent" is an application that runs on the user's device and assists with translation between different languages.

[0497] "Electronic information acquisition means" refers to functions for collecting digital data through terminals, and is responsible for obtaining necessary information from applications and websites.

[0498] "Digital data" refers to electronically processed information such as text, images, and audio.

[0499] An "information processing device" is a device that has the function of analyzing collected digital data and converting it into a translatable format.

[0500] "Generative AI" refers to artificial intelligence technology that learns from large amounts of data and provides appropriate responses and processing results in response to input.

[0501] An "information conversion means" is a device or process that has the function of translating information analyzed using generative AI into multiple languages.

[0502] "Emotional analysis tools" are devices or algorithms that analyze the user's emotions and circumstances, and adjust the system's response and translation content based on that information.

[0503] "Display control device means" refers to a device or process for appropriately displaying translated data on the user's terminal.

[0504] This invention is a multilingual translation system that enables users to communicate smoothly between different languages ​​and provides information based on emotions.

[0505] The user launches a dedicated translation agent on their device. This agent retrieves digital data, including text and images, from sources such as applications and websites. Text information within images is extracted using optical character recognition (OCR) technology installed on the device. Standard OCR software can be used for this process.

[0506] Digital data collected by the terminal is sent to a server. The server analyzes the received data through an information processing device and converts it into a translatable format. Using a generative AI model, the data is translated into multiple languages. In this process, efficient translation is performed using GPT and other language processing AI.

[0507] The translated information is then appropriately adjusted according to the user's emotional state using sentiment analysis tools. This adjustment generates information with a tone and expression that matches the user's emotions. For example, if the user is anxious, the information will be presented concisely, while users seeking a more detailed understanding will be provided with more detailed information.

[0508] Ultimately, the translated data from the server is displayed appropriately on the terminal by the display control device. This allows users to receive information in their native language in an emotionally appropriate manner.

[0509] For example, when a tourist visiting Japan uses a translation agent to make a restaurant reservation, if the sentiment analysis system detects that the user is in a hurry, the process will be simplified. Also, if the sentiment analysis system determines that the user is seeking more information while viewing a tourist guide, additional detailed tourist information will be provided.

[0510] An example of a prompt message would be, "Please explain how the emotion engine works to optimize information delivery when tourists are researching information about Japanese tourist destinations." In this way, this system enables a personalized information experience that goes beyond the scope of language translation.

[0511] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0512] Step 1:

[0513] The user activates the translation agent on their device. This action initiates the entire system process. The input consists of the type and format of information to be translated, specified by the user. The output is that the agent is activated and ready to collect data. Specifically, the user interface opens, and the system awaits further instructions from the user.

[0514] Step 2:

[0515] The device collects digital data from user-specified information sources (e.g., websites, applications). Input is the URL or file path of the information source, and output is extracted as text or image data. In particular, for image data, OCR technology is used to extract text information. Through this specific operation, the device provides raw data in an easy-to-use digital format.

[0516] Step 3:

[0517] The terminal transmits the collected digital data to the server. The input is digital data stored within the terminal, which is converted into a format that the server can process. As output, the data transmitted via a secure protocol is stored on the server. Specifically, the SSL / TLS protocol is used to transmit the data while maintaining its confidentiality and integrity.

[0518] Step 4:

[0519] The server analyzes the received digital data and converts it into a translatable format. The input is the transmitted digital data, and the output is structured data, parsed individually for each language. Specifically, a language detection algorithm is applied, and the data is converted into an appropriate data model.

[0520] Step 5:

[0521] The server translates information analyzed using a generative AI model into multiple languages. The input is structured data, and the output is text translated into multiple target languages. Specifically, the AI ​​model performs natural translation that takes context and nuance into account.

[0522] Step 6:

[0523] The server performs sentiment analysis and adjusts the translated information according to the user's emotions. The input is the user's action history and reactions, and the output is translated data with optimized tone and expression. Specifically, it uses sentiment recognition software to select the most appropriate expression based on the user's situation.

[0524] Step 7:

[0525] The server sends the final translation result to the terminal. The input is the adjusted translation data, and the output is the information displayed on the terminal. Specifically, the data is sent back using the HTTP protocol and converted into a format suitable for display.

[0526] Step 8:

[0527] The terminal displays the received data through a user interface. The input is translated data sent from the server, and the output is information that the user interprets in a way that suits their emotions. Specifically, the translated content is presented on the GUI with the most appropriate font and layout.

[0528] (Application Example 2)

[0529] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0530] When users access information in different languages, there is a challenge in providing more personalized and emotionally relevant information across language barriers. Furthermore, adjusting the amount of information and the tone of translation to suit the user's emotional state has not been adequately done in the past. This limits the improvement of the user experience.

[0531] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0532] In this invention, the server includes means for collecting text data and image data, means for converting them into a translatable format, and means for integrating the translated data. This makes it possible to provide multilingual information tailored to the emotional state of the user.

[0533] A "translation agent" is a program that runs on the user's device and performs translation between multiple languages.

[0534] An "electronic information acquisition means" is an interface for acquiring electronic information such as text data and image data.

[0535] A "server processing means" is a computer system that processes collected data to analyze it and convert it into a translatable format.

[0536] A "translation execution method" is a program that has the function of translating data analyzed using a generative AI into multiple languages.

[0537] "Display control means" refers to a function that transmits translated data to the user's terminal and controls it so that it is properly integrated into the display interface.

[0538] "Emotional analysis means" refers to a function that analyzes the user's emotional state in real time and adjusts the translation content based on the results.

[0539] The system for carrying out this invention consists of a user terminal, a server, a translation agent, and sentiment analysis means. The server receives text and image data collected from the terminal and analyzes it. In doing so, optical character recognition technology is used to convert the collected data into a translatable format. The translation agent translates the data into multiple languages ​​using a generative AI model. The server also analyzes the user's emotional state in real time via the sentiment analysis means and adjusts the translation content based on the results. This adjustment is reflected in the level of detail and tone of the information.

[0540] Ultimately, the server sends the translated data to the terminal, and the display control system shows an integrated interface tailored to the user's emotions. This allows the user to use information in a more personalized and satisfying way.

[0541] A concrete example is a robotic guide at a tourist destination. This guide provides appropriate translations based on the content of the tourist's questions and their emotions. For example, when a tourist asks for a detailed explanation of a sculpture they are interested in, the system detects the tourist's excitement using emotion analysis and provides information accordingly in their native language.

[0542] An example of a prompt is, "Provide detailed information about this sculpture in the native language of interested visitors." This example demonstrates how a generative AI model can be used to provide information tailored to the user's emotions.

[0543] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0544] Step 1:

[0545] The user's device launches the translation agent. The device uses electronic data acquisition means to collect text and image data specified by the user. The input data includes information that the user is interested in. The output is raw text and image data that is sent to the server.

[0546] Step 2:

[0547] The server analyzes the received data. Here, optical character recognition technology is used to extract character information from the image data. The input is the text and image data collected in step 1. As a result of the data processing, text data in a translatable format is output.

[0548] Step 3:

[0549] The server uses a generated AI model to translate translatable text data into multiple specified languages. The input is the translatable text data obtained in step 2. As a result of the translation, multilingual text data is output.

[0550] Step 4:

[0551] The server uses sentiment analysis tools to analyze the user's emotional state based on the information and actions the user has entered. Inputs include data such as user action logs and audio feedback. Outputs are metadata indicating the user's emotional state.

[0552] Step 5:

[0553] The server adjusts the translated multilingual data based on the emotional state. Specifically, it provides a concise translation to users in a hurry and a detailed translation to users who are showing interest. The input is the translated data from step 3 and the emotional metadata from step 4. The output is the adjusted multilingual data.

[0554] Step 6:

[0555] The server sends the adjusted translation data to the user's terminal and integrates it appropriately on the interface using display control means. The terminal displays information that matches the user's emotions. The input is the multilingual data adjusted in step 5, and the output is the display interface on the user's terminal.

[0556] Step 7:

[0557] The user receives the displayed information and requests additional information or provides feedback as needed. This serves as foundational data for providing further information and making adjustments. The input in this step is the displayed information, and the output is new input and feedback from the user.

[0558] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0559] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0560] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0561] [Fourth Embodiment]

[0562] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0563] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0564] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0565] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0566] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0567] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0568] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0569] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0570] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0571] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0572] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0573] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0574] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0575] The system of this invention was developed to improve user convenience and reduce the cost and effort of multilingual support. In this system, a translation agent runs on the user's terminal and collects and translates information from specified applications and websites to provide the user with information in their native language.

[0576] The terminal acquires screen data from applications and websites accessed by the user and sends it to the server in a specific format. The server analyzes the received data and extracts text and image data according to the format. This extracted data is translated into multiple languages ​​by a generative AI. In particular, for image data, optical character recognition technology is used as needed to extract character information and include it in the translation.

[0577] The translated data is sent from the server to the terminal and integrated into the interface in a user-friendly format by display control means. This entire process allows users to easily understand information in their native language and conduct their activities smoothly within Japan.

[0578] For example, when tourists visiting Japan use a local train transfer guidance app, they can simply open the app, and a translation agent will automatically activate, allowing them to view transfer information in their native language. This enables them to travel comfortably without feeling a language barrier, making their sightseeing even more enjoyable.

[0579] This invention is expected to not only expedite the provision of information to tourists but also significantly reduce the burden of multilingual support on public transportation and service providers within Japan. In this way, the present invention will improve the efficiency of multilingual support and contribute to building a win-win relationship between foreign visitors to Japan and Japanese tourism businesses.

[0580] The following describes the processing flow.

[0581] Step 1:

[0582] The user launches the translation agent on their device and opens the application or website to be translated. The device automatically captures the content on the screen and extracts the text and image data that needs to be translated.

[0583] Step 2:

[0584] The terminal sends the extracted text and image data to the server as an analysis request. This request also includes the user's desired target language for translation.

[0585] Step 3:

[0586] The server analyzes the received data, organizing the text and extracting character information from images. Using optical character recognition (OCR) technology, it identifies characters within images and converts them into a translatable format.

[0587] Step 4:

[0588] The server uses generative AI to translate the analyzed text and image data into the specified language. During translation, adjustments are made to generate the most appropriate expression based on the context.

[0589] Step 5:

[0590] The server sends the translated data to the terminal. The terminal then uses this translation to reconfigure the application or website interface in its native language.

[0591] Step 6:

[0592] Users can view the translation results displayed on their device and obtain the necessary information. This allows them to use apps and websites without feeling any language barriers.

[0593] (Example 1)

[0594] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0595] In today's world, where multilingualism is essential, understanding information provided in different languages ​​poses a significant obstacle for users of information processing devices who are not proficient in those languages. Furthermore, information providers face the challenge of incurring substantial costs for multilingual support.

[0596] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0597] In this invention, the server includes means for a language conversion agent activated on the user's information processing device to collect textual and visual information via a target content acquisition means, data processing means for analyzing the collected textual and visual information and converting it into a language-convertible format, and conversion execution means for converting the analyzed data into multiple languages ​​using a generation AI. This allows users to easily obtain information in their native language and reduces the costs associated with multilingual support.

[0598] An "information processing device" is an electronic device that allows users to acquire and process information via the internet or applications.

[0599] A "language conversion agent" is a software program that operates on an information processing device and collects textual and visual information from content accessed by the user.

[0600] "Content acquisition means" refers to the process or technology used by an information processing device to acquire necessary data from applications or websites it accesses.

[0601] "Textual information" refers to a form of information recorded as text data, such as strings of characters found in documents and emails.

[0602] "Visual information" refers to data provided in a visual format, including elements found in images and videos.

[0603] "Data processing means" refers to processes and equipment that analyze collected textual and visual information and convert it for use in other formats or applications.

[0604] "Generative AI" is a system that uses artificial intelligence technology to generate new information or output from input data according to a specified task.

[0605] "Optical character recognition technology" is a technology that converts characters contained in visual information into electronic text data.

[0606] The system of this invention takes the form of a language conversion agent operating within an information processing device in order to improve user convenience. The information processing device acquires content from applications and websites accessed by the user and converts it into a standardized data format. In this acquisition process, text and image data are collected using screen capture technology and APIs.

[0607] The server receives data acquired from the terminal and performs analysis using its built-in data processing capabilities. During the analysis, textual information is distinguished from visual information, and the visual information is converted into text using optical character recognition technology (e.g., open-source OCR systems). This process needs to be particularly fast and accurate in applications requiring multilingual support.

[0608] The analyzed data is then translated into the specified language by the generating AI. The generating AI performs multilingual translation based on the prompt message specified during the language conversion. An example of such a prompt message is, "Translate this text into the target language."

[0609] The translated data is sent from the server to the user's information processing device, and the terminal integrates that data into the user interface using a display management mechanism. For example, when a user sightseeing in Japan uses a local guide app, the terminal automatically displays the translated content, allowing them to check information in real time without experiencing language barriers.

[0610] This allows users to easily understand information provided in different languages ​​in their own language, while also reducing the burden of multilingual support for information providers. Throughout this entire system, convenient and efficient information sharing is achieved for both users and information providers.

[0611] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0612] Step 1:

[0613] The user's device captures screen information from applications and websites selected by the user. The input to this process is the state of the screen the user is viewing, and the output is the captured screenshot or text data. Specifically, the device uses a screen capture API to collect data and saves it in a formatted data format for further processing.

[0614] Step 2:

[0615] The terminal sends the acquired data to the server in a standard format. The input is the screenshot or text data obtained in step 1, and the output is the data packet sent to the server. For security purposes, the terminal encrypts and transmits the data using the HTTPS protocol.

[0616] Step 3:

[0617] The server receives data transmitted from the terminal and performs analysis using data processing tools. The input data consists of images and text information, and the output consists of analyzed text and visual information. Specific processing includes extracting characters from image data using optical character recognition technology.

[0618] Step 4:

[0619] The server translates the data analyzed using a generative AI model. The input is the text data obtained in step 3, and the output is the text translated into the specified language. A prompt is used to perform the translation; for example, entering "Translate this text into English" will perform the translation between multiple languages.

[0620] Step 5:

[0621] The server resends the translated data to the user's device. The input is the translated text, and the output is the display data sent to the device. The server formats the translation results appropriately and sends them to the device via the network protocol.

[0622] Step 6:

[0623] The terminal integrates the received translation data into the user interface and displays it to the user. The input is the translated data received in step 5, and the output is an information screen in the user's native language that the user can view. The terminal adjusts the font size and layout for user readability and presents information that is updated in real time.

[0624] (Application Example 1)

[0625] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0626] When visitors and travelers shop in foreign countries, language barriers can be a significant obstacle in understanding product information. In particular, accurate and rapid translation is essential for accurately understanding product labels and ingredient information. However, traditional methods require considerable effort to implement multilingual support, making it difficult for users. To address this challenge, an effective and efficient means of translating and displaying product information is necessary.

[0627] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0628] In this invention, the server includes means for a translation agent launched on the user's terminal to collect text data and visual data via a digital information acquisition means; computing device processing means for analyzing the collected text data and visual data and converting them into a translatable format; translation execution means for translating the analyzed data into multiple languages ​​using a generation AI; and means for the user to acquire product information via a camera, translate that information into their native language, and provide it to the customer. This enables visitors to instantly understand product information in their native language and make shopping smoother.

[0629] A "translation agent" is a software module that runs on the user's device and automatically collects and translates digital information.

[0630] "Means of acquiring digital information" refers to an interface or technology for acquiring text data and visual data through a terminal.

[0631] "Computational device processing means" refers to a computing device or process for analyzing collected data and converting it into a translatable format.

[0632] "Generative AI" refers to artificial intelligence models used to translate input data into multiple languages.

[0633] "Translation execution means" refers to the hardware or software process necessary to actually translate the analyzed data.

[0634] A "display interface" is a screen or application used to visually present translated information to the user.

[0635] "Visual data" is a general term for information that can be recognized visually, such as images and videos.

[0636] "Optical character recognition technology" is a technology for extracting character information from images.

[0637] "Product information" refers to detailed descriptions of the product, such as price and ingredients.

[0638] This invention constructs a system that provides product information to users in multiple languages ​​using a translation agent launched on the user's terminal. The terminal has a built-in camera, which is used to acquire product information in the store as visual data. The visual data is collected by a digital information acquisition means on the terminal, analyzed by a computing device processing means, and then converted into a translatable format. Subsequently, the analyzed data is translated using a generation AI and transmitted to the user's terminal and integrated into the display interface.

[0639] This process utilizes optical character recognition (OCR) technologies such as the Google Cloud Vision API to extract textual information from visual data and treat it as the target for translation. External translation services such as the Google Translate API can be used for translation. Furthermore, AI models such as OpenAI GPT are used as the generative AI model to provide customers with natural and contextually relevant translation results.

[0640] As a concrete example, when a foreign tourist picks up an item in a souvenir shop, they use their smartphone camera to focus on the product label. At that moment, OCR technology recognizes the text information, instantly translates it into the desired language, and displays it on the smartphone screen. As an example of a prompt sentence in this process, the following sentence is provided to the generating AI model: "Translate the following Japanese product label text into English: 'Matcha (Uji Tea) 100g'. Provide a consumer-friendly description of the tea's taste and usage in a comprehensive manner." and the translation is optimized.

[0641] This invention allows users to instantly understand product information in their own language, enabling them to shop smoothly in foreign countries.

[0642] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0643] Step 1:

[0644] The user activates the device's camera and takes a picture of the product label. The captured image data is collected by a digital information acquisition system. The input is image data from the camera, and the device prepares this data to send to the next step.

[0645] Step 2:

[0646] The terminal sends the collected image data to the server. The server uses optical character recognition (OCR) technology to analyze the image data and extract text information. The input is image data, and the output is extracted text data. This text data is ready for translation.

[0647] Step 3:

[0648] The server translates text data extracted using generative AI. The server inputs prompt sentences into the generative AI model and translates text into multiple languages ​​based on instructions such as "Translate the following product label text into English: 'Matcha (Uji Tea) 100g'." The input is text data, and the output is translated text data.

[0649] Step 4:

[0650] The translated text data is sent from the server to the terminal. The terminal uses this data to visually display it to the user and integrate it into the interface. The input is the translated text data, and the output is the translated information displayed on the user's screen. This allows the user to view product information in their native language.

[0651] This series of processes allows users to easily understand and utilize product information in foreign countries.

[0652] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0653] This invention is a system that combines a translation agent with an emotion engine to enable the provision of multilingual information tailored to the user's emotions. The aim is to achieve communication that transcends language barriers and provide users with a more personal and enriching experience.

[0654] At system startup, the user activates the translation agent on their device. The device collects text and image data from specified applications and websites and sends it to the server. This data is converted into a translatable format and translated into the specified language by a generative AI. In particular, text information contained in image data is extracted using optical character recognition technology and targeted for translation.

[0655] The emotion engine, a key feature of this invention, analyzes user actions and inputs in real time to recognize the user's emotions. For example, if a user is urgently seeking information, the engine detects this urgency and presents the translated content concisely and intuitively. Conversely, in situations where the user is taking their time to review the information, the engine adjusts to provide a more detailed translation.

[0656] The translated data is sent from the server to the terminal and displayed on the interface using display control means, with a tone and expression that matches the user's emotions. In this way, users can receive translated content in their native language in a way that suits their emotions, enabling a more natural and satisfying information experience.

[0657] For example, when a tourist visiting Japan tries to make a restaurant reservation, if the emotional engine recognizes their impatience, it will quickly provide a simplified reservation method, resulting in a smooth reservation experience. Similarly, when a tourist is looking at a travel guide, if the emotional engine recognizes their heightened curiosity and interest, it will provide more detailed information, enriching the experience.

[0658] In this system, the combination of emotion recognition by an emotion engine and multilingual translation by generative AI enables flexible information provision tailored to the user's situation and emotions, playing a role in making information acquisition in Japan more convenient.

[0659] The following describes the processing flow.

[0660] Step 1:

[0661] The user launches the translation agent on their device and opens the desired application or website. The device collects text and image data from the app or website, and the emotion engine detects the user's current emotional state.

[0662] Step 2:

[0663] The device uses an emotion engine to analyze the user's operation speed and screen touch frequency, recognizing the user's emotions in real time. This information is sent to the server along with the data.

[0664] Step 3:

[0665] The server analyzes the received data and organizes the text data. Text information within images is extracted using optical character recognition technology. The server also analyzes the user's sentiment data to determine if appropriate adjustments to the translation are necessary.

[0666] Step 4:

[0667] The server translates the data analyzed using generative AI into the specified language. In doing so, it uses a tone and expression that reflects the recognized user's emotions. For example, it provides a concise translation if the user is in a hurry, and a more detailed translation if they are excited.

[0668] Step 5:

[0669] The server sends the translation results to the terminal. The terminal then presents the user with an adapted interface based on these translation results. The displayed information is provided in a format that harmonizes with the user's perceived emotions.

[0670] Step 6:

[0671] Users can review translations optimized to their emotions and obtain the information they need. This allows for a more natural and less stressful information retrieval experience.

[0672] (Example 2)

[0673] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0674] In today's globalized society, smooth communication between users who speak different languages ​​is essential. However, simply translating languages ​​makes it difficult to provide appropriate information that takes into account the user's feelings and circumstances. Therefore, a system is needed that can provide more personalized and context-appropriate information.

[0675] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0676] In this invention, the server includes an information processing device, an information conversion device, and an emotion analysis device. This makes it possible to provide a multilingual translation service that is adjusted according to the user's emotions and feelings.

[0677] A "translation agent" is an application that runs on the user's device and assists with translation between different languages.

[0678] "Electronic information acquisition means" refers to functions for collecting digital data through terminals, and is responsible for obtaining necessary information from applications and websites.

[0679] "Digital data" refers to electronically processed information such as text, images, and audio.

[0680] An "information processing device" is a device that has the function of analyzing collected digital data and converting it into a translatable format.

[0681] "Generative AI" refers to artificial intelligence technology that learns from large amounts of data and provides appropriate responses and processing results in response to input.

[0682] An "information conversion means" is a device or process that has the function of translating information analyzed using generative AI into multiple languages.

[0683] "Emotional analysis tools" are devices or algorithms that analyze the user's emotions and circumstances, and adjust the system's response and translation content based on that information.

[0684] "Display control device means" refers to a device or process for appropriately displaying translated data on the user's terminal.

[0685] This invention is a multilingual translation system that enables users to communicate smoothly between different languages ​​and provides information based on emotions.

[0686] The user launches a dedicated translation agent on their device. This agent retrieves digital data, including text and images, from sources such as applications and websites. Text information within images is extracted using optical character recognition (OCR) technology installed on the device. Standard OCR software can be used for this process.

[0687] Digital data collected by the terminal is sent to a server. The server analyzes the received data through an information processing device and converts it into a translatable format. Using a generative AI model, the data is translated into multiple languages. In this process, efficient translation is performed using GPT and other language processing AI.

[0688] The translated information is then appropriately adjusted according to the user's emotional state using sentiment analysis tools. This adjustment generates information with a tone and expression that matches the user's emotions. For example, if the user is anxious, the information will be presented concisely, while users seeking a more detailed understanding will be provided with more detailed information.

[0689] Ultimately, the translated data from the server is displayed appropriately on the terminal by the display control device. This allows users to receive information in their native language in an emotionally appropriate manner.

[0690] For example, when a tourist visiting Japan uses a translation agent to make a restaurant reservation, if the sentiment analysis system detects that the user is in a hurry, the process will be simplified. Also, if the sentiment analysis system determines that the user is seeking more information while viewing a tourist guide, additional detailed tourist information will be provided.

[0691] An example of a prompt message would be, "Please explain how the emotion engine works to optimize information delivery when tourists are researching information about Japanese tourist destinations." In this way, this system enables a personalized information experience that goes beyond the scope of language translation.

[0692] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0693] Step 1:

[0694] The user activates the translation agent on their device. This action initiates the entire system process. The input consists of the type and format of information to be translated, specified by the user. The output is that the agent is activated and ready to collect data. Specifically, the user interface opens, and the system awaits further instructions from the user.

[0695] Step 2:

[0696] The device collects digital data from user-specified information sources (e.g., websites, applications). Input is the URL or file path of the information source, and output is extracted as text or image data. In particular, for image data, OCR technology is used to extract text information. Through this specific operation, the device provides raw data in an easy-to-use digital format.

[0697] Step 3:

[0698] The terminal transmits the collected digital data to the server. The input is digital data stored within the terminal, which is converted into a format that the server can process. As output, the data transmitted via a secure protocol is stored on the server. Specifically, the SSL / TLS protocol is used to transmit the data while maintaining its confidentiality and integrity.

[0699] Step 4:

[0700] The server analyzes the received digital data and converts it into a translatable format. The input is the transmitted digital data, and the output is structured data, parsed individually for each language. Specifically, a language detection algorithm is applied, and the data is converted into an appropriate data model.

[0701] Step 5:

[0702] The server translates information analyzed using a generative AI model into multiple languages. The input is structured data, and the output is text translated into multiple target languages. Specifically, the AI ​​model performs natural translation that takes context and nuance into account.

[0703] Step 6:

[0704] The server performs sentiment analysis and adjusts the translated information according to the user's emotions. The input is the user's action history and reactions, and the output is translated data with optimized tone and expression. Specifically, it uses sentiment recognition software to select the most appropriate expression based on the user's situation.

[0705] Step 7:

[0706] The server sends the final translation result to the terminal. The input is the adjusted translation data, and the output is the information displayed on the terminal. Specifically, the data is sent back using the HTTP protocol and converted into a format suitable for display.

[0707] Step 8:

[0708] The terminal displays the received data through a user interface. The input is translated data sent from the server, and the output is information that the user interprets in a way that suits their emotions. Specifically, the translated content is presented on the GUI with the most appropriate font and layout.

[0709] (Application Example 2)

[0710] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0711] When users access information in different languages, there is a challenge in providing more personalized and emotionally relevant information across language barriers. Furthermore, adjusting the amount of information and the tone of translation to suit the user's emotional state has not been adequately done in the past. This limits the improvement of the user experience.

[0712] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0713] In this invention, the server includes means for collecting text data and image data, means for converting them into a translatable format, and means for integrating the translated data. This makes it possible to provide multilingual information tailored to the emotional state of the user.

[0714] A "translation agent" is a program that runs on the user's device and performs translation between multiple languages.

[0715] An "electronic information acquisition means" is an interface for acquiring electronic information such as text data and image data.

[0716] A "server processing means" is a computer system that processes collected data to analyze it and convert it into a translatable format.

[0717] A "translation execution method" is a program that has the function of translating data analyzed using a generative AI into multiple languages.

[0718] "Display control means" refers to a function that transmits translated data to the user's terminal and controls it so that it is properly integrated into the display interface.

[0719] "Emotional analysis means" refers to a function that analyzes the user's emotional state in real time and adjusts the translation content based on the results.

[0720] The system for carrying out this invention consists of a user terminal, a server, a translation agent, and sentiment analysis means. The server receives text and image data collected from the terminal and analyzes it. In doing so, optical character recognition technology is used to convert the collected data into a translatable format. The translation agent translates the data into multiple languages ​​using a generative AI model. The server also analyzes the user's emotional state in real time via the sentiment analysis means and adjusts the translation content based on the results. This adjustment is reflected in the level of detail and tone of the information.

[0721] Ultimately, the server sends the translated data to the terminal, and the display control system shows an integrated interface tailored to the user's emotions. This allows the user to use information in a more personalized and satisfying way.

[0722] A concrete example is a robotic guide at a tourist destination. This guide provides appropriate translations based on the content of the tourist's questions and their emotions. For example, when a tourist asks for a detailed explanation of a sculpture they are interested in, the system detects the tourist's excitement using emotion analysis and provides information accordingly in their native language.

[0723] An example of a prompt is, "Provide detailed information about this sculpture in the native language of interested visitors." This example demonstrates how a generative AI model can be used to provide information tailored to the user's emotions.

[0724] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0725] Step 1:

[0726] The user's device launches the translation agent. The device uses electronic data acquisition means to collect text and image data specified by the user. The input data includes information that the user is interested in. The output is raw text and image data that is sent to the server.

[0727] Step 2:

[0728] The server analyzes the received data. Here, optical character recognition technology is used to extract character information from the image data. The input is the text and image data collected in step 1. As a result of the data processing, text data in a translatable format is output.

[0729] Step 3:

[0730] The server uses a generated AI model to translate translatable text data into multiple specified languages. The input is the translatable text data obtained in step 2. As a result of the translation, multilingual text data is output.

[0731] Step 4:

[0732] The server uses sentiment analysis tools to analyze the user's emotional state based on the information and actions the user has entered. Inputs include data such as user action logs and audio feedback. Outputs are metadata indicating the user's emotional state.

[0733] Step 5:

[0734] The server adjusts the translated multilingual data based on the emotional state. Specifically, it provides a concise translation to users in a hurry and a detailed translation to users who are showing interest. The input is the translated data from step 3 and the emotional metadata from step 4. The output is the adjusted multilingual data.

[0735] Step 6:

[0736] The server sends the adjusted translation data to the user's terminal and integrates it appropriately on the interface using display control means. The terminal displays information that matches the user's emotions. The input is the multilingual data adjusted in step 5, and the output is the display interface on the user's terminal.

[0737] Step 7:

[0738] The user receives the displayed information and requests additional information or provides feedback as needed. This serves as foundational data for providing further information and making adjustments. The input in this step is the displayed information, and the output is new input and feedback from the user.

[0739] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0740] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0741] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0742] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0743] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0744] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0745] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0746] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0747] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0748] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0749] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0750] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0751] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0752] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0753] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0754] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0755] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0756] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0757] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0758] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0759] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0760] The following is further disclosed regarding the embodiments described above.

[0761] (Claim 1)

[0762] A translation agent launched on the user's terminal collects text data and image data via a target electronic information acquisition means,

[0763] A server processing means that analyzes collected text data and image data and converts them into a translatable format,

[0764] A translation execution means that translates data analyzed using generative AI into multiple languages,

[0765] A display control means that transmits the translated data to the user's terminal and integrates it into the display interface,

[0766] A system that includes this.

[0767] (Claim 2)

[0768] The system according to claim 1, wherein the generating AI analyzes and translates character information contained in image data using optical character recognition technology.

[0769] (Claim 3)

[0770] The system according to claim 1, wherein the server processing means obtains the language desired by the user and optimizes the translation based on that language.

[0771] "Example 1"

[0772] (Claim 1)

[0773] A language conversion agent activated on the user's information processing device collects textual and visual information via a target content acquisition means,

[0774] A data processing means for analyzing collected textual and visual information and converting it into a language-convertible format,

[0775] A conversion execution means that converts data analyzed using a generative AI into multiple languages,

[0776] A display management means that transmits the converted data to the user's information processing device and integrates it into the display interface,

[0777] A system that includes this.

[0778] (Claim 2)

[0779] The system according to claim 1, wherein a generating AI analyzes character information contained in visual information using optical character recognition technology and converts it into language.

[0780] (Claim 3)

[0781] The system according to claim 1, wherein the data processing means obtains the language desired by the user and optimizes language conversion based on that language.

[0782] "Application Example 1"

[0783] (Claim 1)

[0784] A translation agent launched on the user's device collects text data and visual data via a means for acquiring the target digital information,

[0785] A computing device processing means that analyzes collected text data and visual data and converts it into a translatable format,

[0786] A translation execution means that translates data analyzed using generative AI into multiple languages,

[0787] A display control means that transmits the translated data to the user's terminal and integrates it into the display interface,

[0788] A means by which users acquire product information via camera, translate that information into their own language, and provide it to customers,

[0789] A system that includes this.

[0790] (Claim 2)

[0791] The system according to claim 1, wherein a generating AI analyzes and translates character information contained in visual data using optical character recognition technology.

[0792] (Claim 3)

[0793] The system according to claim 1, wherein the computing device processing means obtains the language desired by the user and optimizes the translation based on that language.

[0794] "Example 2 of combining an emotion engine"

[0795] (Claim 1)

[0796] A translation agent launched on the user's terminal collects digital data via a means for acquiring the target electronic information,

[0797] Information processing device means for analyzing collected digital data and converting it into a translatable format,

[0798] An information conversion means that translates information analyzed using generative AI into multiple languages,

[0799] A means of adjusting the translation result based on the user's emotions using emotion analysis means,

[0800] A display control device means that transmits the translated data to the user's terminal and integrates it into a display device,

[0801] A system that includes this.

[0802] (Claim 2)

[0803] The system according to claim 1, wherein a generating AI analyzes and translates character information contained in digital data using recognition technology.

[0804] (Claim 3)

[0805] The system according to claim 1, wherein the information processing device acquires the language desired by the user and optimizes the translation based on that language.

[0806] "Application example 2 when combining with an emotional engine"

[0807] (Claim 1)

[0808] A translation agent launched on the user's terminal collects text data and image data via a target electronic information acquisition means,

[0809] A server processing means that analyzes collected text data and image data and converts them into a translatable format,

[0810] A translation execution means that translates data analyzed using generative AI into multiple languages,

[0811] A display control means that transmits the translated data to the user's terminal and integrates it into the display interface,

[0812] A sentiment analysis tool that analyzes the user's emotional state in real time and adjusts the translation content accordingly,

[0813] A system that includes this.

[0814] (Claim 2)

[0815] The system according to claim 1, wherein the generating AI analyzes and translates character information contained in image data using optical character recognition technology.

[0816] (Claim 3)

[0817] The system according to claim 1, wherein the server processing means obtains the language desired by the user and optimizes the translation based on that language. [Explanation of symbols]

[0818] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A translation agent launched on the user's terminal collects text data and image data via a target electronic information acquisition means, A server processing means that analyzes collected text data and image data and converts them into a translatable format, A translation execution means that translates data analyzed using generative AI into multiple languages, A display control means that transmits the translated data to the user's terminal and integrates it into the display interface, A system that includes this.

2. The system according to claim 1, wherein the generating AI analyzes and translates character information contained in image data using optical character recognition technology.

3. The system according to claim 1, wherein the server processing means obtains the language desired by the user and optimizes the translation based on that language.

Images