system

Abstract

Provide a system. [Solution] A means for acquiring location information; A means for searching and generating surrounding information based on the acquired location information; means for displaying the generated information; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, it is important for users to instantly obtain the information they need and communicate efficiently. However, conventional technologies require the use of multiple devices and applications to meet these needs, resulting in issues such as complex operation and delays in information acquisition. Furthermore, it has been difficult to realize a wide range of functions, such as foreign language translation, on-site work support, customer service optimization, and detection and warning of dangerous situations, in a single device. For this reason, there is a demand for devices that are both multifunctional and intuitive to operate. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means. That is, a system including a means for acquiring location information, a means for searching for and generating surrounding information based on the acquired location information, and a means for displaying the generated information. The system may further include a means for capturing audio in different languages, a means for translating the captured audio into a specified language, and a means for providing the translated audio. It may also include a means for capturing video of the surroundings using a 360-degree camera, a means for analyzing the captured video to detect dangerous situations, and a means for alerting the user to the detected dangerous situations. The system may further include a means for acquiring work location information, a means for searching for and displaying related work procedures based on the acquired location information, a means for capturing conversations with customers, a means for analyzing the captured conversations to generate optimal response examples, and a means for providing the generated response examples. It may also include a virtual reality mode for conducting English conversation lessons, a means for capturing and analyzing conversations between a user and a virtual avatar, and a means for generating and providing responses from the virtual avatar based on the analysis results.

[0006] "Location information" is data for identifying the user's current geographical location.

[0007] "Nearby information" is data about places and events that are near the user's current location and that may be of interest or concern to the user.

[0008] "Searching" is the process of locating specific information from a database or source.

[0009] "Generation" is the process of creating results based on data and information to present to the user.

[0010] "Display" means providing acquired information or data in a form that can be visually confirmed by the user.

[0011] "Audio capture" refers to the use of a device such as a microphone to capture an audio signal and process it as digital data.

[0012] "Translation" is the process of transcribing text or audio expressed in one language into a different language.

[0013] A "360-degree camera" is a device that can capture an image of the entire surroundings at once.

[0014] "Analysis" is the process of examining acquired data in detail to understand its meaning and derive results.

[0015] A "warning" is a notification or alert that notifies the user of a potential danger or problem.

[0016] A "work procedure" is a list of steps to be followed to efficiently complete a particular task or operation.

[0017] "Conversation with a customer" refers to communication between a service provider and a customer.

[0018] An "optimal response example" is the most appropriate response or service delivery method for a customer's request or question.

[0019] "Virtual reality mode" is a technology that immerses users in a computer-generated virtual environment.

[0020] A "virtual avatar" is a digital character designed to interact with a user within a virtual reality environment. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] The smart ARVR glasses system of the present invention is a multi-functional device that allows users to obtain various information in real time and provide efficient communication and work assistance.

[0043] System configuration

[0044] The system consists of the following main components:

[0045] 1. Device (Smart ARVR Glasses)

[0046] GPS sensor

[0047] 360-degree camera

[0048] microphone

[0049] speaker

[0050] Display (AR / VR mode switchable)

[0051] Network Communication Module

[0052] 2. Server

[0053] Database Management Systems

[0054] Translation Engine

[0055] Video Analysis Engine

[0056] Various information services such as weather forecasts, restaurant information, and tourist information

[0057] Information acquisition and provision process

[0058] When a user puts on the smart ARVR glasses and begins using them, the device first obtains its current location using the GPS sensor. This location information is then sent to the server via the network communication module. The server then uses the received location information to search its database for information on nearby restaurants, tourist spots, dangerous roads, weather forecasts, and other information, generating optimized information. The generated information is then sent to the device and provided to the user via voice or an AR screen.

[0059] Machine translation function

[0060] When a user starts a conversation with someone who speaks a foreign language, the device's microphone captures the voice and sends the captured voice data to the server. The server then passes the voice data through a translation engine to translate it into the required language. The translation result is then sent to the device and provided to the user as audio or subtitles.

[0061] Check road conditions in real time and avoid accidents

[0062] While the user is moving, the device's 360-degree camera captures video of the surroundings in real time. This video data is sent to a server, which uses a video analytics engine to detect dangerous situations or potential accidents. The results are then sent to the device, and a warning is displayed to the user via audio and an AR screen.

[0063] Work support

[0064] When a user arrives at a designated work area at a manufacturing or logistics site, the device sends location information to a server. The server searches a database for work procedures related to that area based on the location information, and sends the retrieved work procedures to the device. The device then displays the procedures in AR within the user's field of view, helping to improve work efficiency.

[0065] Customer service support

[0066] When a user starts a conversation with a customer while serving them, the device uses its voice recognition function to capture the conversation. The captured conversation data is sent to a server, where an analysis engine generates the most appropriate response. The generated response is sent to the device and provided to the user as audio or subtitles.

[0067] English conversation lessons

[0068] When a user starts an English conversation lesson, the device switches to VR mode and displays a generated AI model avatar. The user converses with the avatar, and their utterances are captured by the device and sent to the server. The server analyzes the utterances, generates appropriate responses, and sends them to the device. The device then provides the generated responses to the user through the avatar.

[0069] Specific examples

[0070] When a user is walking near Shinjuku Station, the device acquires location information and sends it to the server. The server searches for recommended restaurants and weather information around Shinjuku Station, generates the most appropriate information, and sends it to the device. The device then announces, "Here are some recommended restaurants," and displays it on the AR screen.

[0071] In addition, if a user is conversing in English with a foreign tourist, the device captures the conversation and sends it to the server. The server translates the English into Japanese and sends the translation result to the device. The device displays the translated Japanese as subtitles, allowing the user to communicate smoothly.

[0072] As described above, the smart ARVR glasses system of the present invention can provide users with a wide range of information support in real time by integrating multiple functions such as location information, voice capture, translation, and video analysis.

[0073] The processing flow will be explained below.

[0074] Information acquisition and provision process

[0075] Step 1:

[0076] Device: Uses the GPS sensor to obtain current location information.

[0077] Step 2:

[0078] Terminal: Sends the acquired location information to the server via the network communication module.

[0079] Step 3:

[0080] Server: Based on the received location information, the server searches a database for information such as nearby restaurants, tourist spots, dangerous roads, and weather forecasts.

[0081] Step 4:

[0082] Server: Optimizes search results and generates the information provided.

[0083] Step 5:

[0084] Server: Sends the generated information to the terminal.

[0085] Step 6:

[0086] Terminal: Provides received information to the user via voice or an AR screen.

[0087] Machine translation function

[0088] Step 1:

[0089] User: Start a conversation with someone who speaks a foreign language.

[0090] Step 2:

[0091] Device: Uses the microphone to capture audio.

[0092] Step 3:

[0093] Device: Sends captured audio data to the server.

[0094] Step 4:

[0095] Server: The received voice data is passed through a translation engine and translated into the specified language.

[0096] Step 5:

[0097] Server: Sends the translation results to the device.

[0098] Step 6:

[0099] Device: Providing translated audio and subtitles to users.

[0100] Check road conditions in real time and avoid accidents

[0101] Step 1:

[0102] Device: Uses a 360-degree camera to capture footage of the surroundings in real time.

[0103] Step 2:

[0104] Terminal: Sends captured video data to the server.

[0105] Step 3:

[0106] Server: Analyzes the received video data and detects dangerous situations and potential accidents.

[0107] Step 4:

[0108] Server: Sends the detection results to the device.

[0109] Step 5:

[0110] Device: Provides warning information to users via voice or AR screen.

[0111] Work support

[0112] Step 1:

[0113] User: Arrives at a designated work area on a manufacturing or logistics site.

[0114] Step 2:

[0115] Device: Obtains current location information and sends it to the server.

[0116] Step 3:

[0117] Server: Based on the location information, it searches the database for work procedures relevant to that area.

[0118] Step 4:

[0119] Server: Sends search results to the device.

[0120] Step 5:

[0121] Terminal: Work procedures are displayed in AR in the user's field of view.

[0122] Customer service support

[0123] Step 1:

[0124] User: Starts a conversation with a customer.

[0125] Step 2:

[0126] Device: Use voice recognition to capture what you say.

[0127] Step 3:

[0128] Terminal: Sends captured conversation data to the server.

[0129] Step 4:

[0130] Server: Analyzes conversation data and generates optimal response examples.

[0131] Step 5:

[0132] Server: Sends the generated response examples to the device.

[0133] Step 6:

[0134] Device: Provide users with examples of what works with audio and subtitles.

[0135] English conversation lessons

[0136] Step 1:

[0137] User: Switches to VR mode to begin an English conversation lesson.

[0138] Step 2:

[0139] Terminal: Display the generated AI model avatar.

[0140] Step 3:

[0141] User: Converse with an AI avatar.

[0142] Step 4:

[0143] Terminal: Captures what the user says and sends it to the server.

[0144] Step 5:

[0145] Server: Analyzes what the user says and generates an appropriate response.

[0146] Step 6:

[0147] Server: Generates a response and sends it to the device.

[0148] Step 7:

[0149] Terminal: The generated response is presented to the user through an avatar.

[0150] Example 1

[0151] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0152] In conventional smart devices and information systems, functions such as location-based information provision, language translation, and hazard detection were implemented separately, forcing users to use multiple devices and applications simultaneously. Furthermore, because the various functions were not linked in real time, it was difficult to efficiently obtain information or provide work support. Therefore, a system that integrated all functions was needed.

[0153] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0154] In this invention, the server includes means for acquiring location information, means for searching for surrounding information based on the acquired location information, means for generating optimal information based on the searched information, means for displaying the generated information, means for capturing audio in different languages, means for translating the captured audio into a specified language, means for providing the translated result, means for capturing video, means for analyzing the captured video to detect a dangerous situation, means for alerting the user of detected danger information, means for inputting a prompt sentence, means for analyzing the input prompt sentence, means for generating an appropriate response based on the analysis result, and means for providing the generated response. This enables a single system to provide multiple functions including information provision, translation, danger detection, and conversation support based on prompts.

[0155] "Means for acquiring location information" refers to a device or system for measuring the current location and acquiring that information as digital data.

[0156] The "means for searching surrounding information" is a system for extracting information about the region and environment from a database based on the acquired location information.

[0157] "Means for generating optimal information" refers to algorithms or programs that evaluate the searched information, convert it into the most useful form for the user, and provide it to the user.

[0158] The "means for displaying information" refers to a device or interface for visually or audibly presenting the generated information to the user.

[0159] A "means for capturing audio in different languages" is a system for using a device such as a microphone to pick up audio spoken in different languages ​​and record it as digital data.

[0160] A "means for translating speech into a specified language" is software or algorithms that analyze the captured speech data and translate it into a target language.

[0161] "Means for providing translated results" refers to a function or device for outputting translated language data to the user in the form of voice or text.

[0162] A "means for capturing video" is a system that uses a device such as a camera to capture video of the surroundings and record it as digital data.

[0163] "Means for analyzing captured video to detect dangerous situations" refers to algorithms or programs that analyze the acquired video data and recognize dangerous situations or abnormal movements from it.

[0164] The "means for warning the user about danger information" refers to means such as sound, visual display, vibration, etc. for notifying the user of the detected danger information.

[0165] A "means for inputting prompt sentences" is an interface that allows the user to freely enter questions or instructions into the system through text input or voice input.

[0166] The "means for analyzing the input prompt sentence" is a natural language processing algorithm for understanding the prompt sentence input by the user and performing appropriate processing according to its content.

[0167] The "means for generating an appropriate response" is software or a model for generating an accurate and appropriate response to the user based on the analyzed prompt sentence.

[0168] The "means for providing a response" refers to a function or device that outputs the generated response to the user in voice or text form and supports the dialogue with the user.

[0169] The system of the present invention is an integrated information provision system that combines a smart device and a server. This makes it possible to provide multiple functions such as location information, translation, road condition confirmation, and prompt sentence analysis in an integrated manner. The specific configuration and operation procedure are described below.

[0170] System configuration

[0171] The system mainly consists of the following components:

[0172] 1. Terminal (smart device such as smart glasses)

[0173] GPS sensor

[0174] Cameras (including 360-degree cameras)

[0175] microphone

[0176] speaker

[0177] Display (AR / VR mode switchable)

[0178] Network communication module (Wi-Fi and mobile data communication)

[0179] 2. Server

[0180] Database Management Systems

[0181] Translation engine (e.g., Google Translate API)

[0182] Video Analysis Engine

[0183] information search engine

[0184] Generative AI Models

[0185] Main functions and processing flow

[0186] Information provision function

[0187] When a user puts on a smart device and starts using it, the device's GPS sensor acquires its current location. The acquired location information is sent to a server via a network communication module. Based on this location information, the server uses a database management system to search for information about the surrounding area and generate the most appropriate information from the search results. This information is then provided to the user in the form of audio, AR display, or other formats.

[0188] Specific examples

[0189] When a user is walking near Shinjuku Station, the device's GPS sensor acquires location information and sends it to the server. The server searches for recommended restaurants and weather information around Shinjuku Station, generates the most appropriate information, and sends it to the device. The device then announces, "Here are some recommended restaurants," and displays it on the AR screen.

[0190] Machine translation function

[0191] When a user starts a conversation with someone speaking a foreign language, the device's microphone captures the audio and sends it to the server. The server then translates the audio data into the required language through a translation engine and sends the translation results to the device. The translation results are then provided to the user as audio or subtitles.

[0192] Specific examples

[0193] When a user is conversing in English with a foreign tourist, the device's microphone captures the conversation and sends it to the server. The server then passes the voice data through a translation engine to translate the English into Japanese and sends the result to the device. The device then displays the translated Japanese as subtitles, allowing the user to communicate smoothly.

[0194] Road condition check function

[0195] While the user is moving, the device's 360-degree camera captures video of the surroundings. This video data is sent to a server, which uses a video analysis engine to detect dangerous situations or potential accidents. The results are then sent to the device, where they are used to warn the user via audio and AR displays.

[0196] Prompt sentence analysis function

[0197] When a user enters a prompt, the device sends the text or voice data to the server, which uses a generative AI model to analyze the prompt and generate an appropriate response, which is then sent to the device and provided to the user as voice or text.

[0198] Prompt Sentence Examples

[0199] "Please tell me some recommended restaurants near Shinjuku Station."

[0200] As described above, the system of the present invention provides multiple functions, including location information, voice capture, translation, video analysis, and prompt sentence analysis, in an integrated manner, and can provide multifunctional information support to users in real time.

[0201] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0202] Step 1:

[0203] The user wears a smart device and starts the system. The device acquires the current location information using the GPS sensor. This location information (input) is stored on the device as coordinate data (output).

[0204] Step 2:

[0205] The device sends the acquired location information to the server via the network communication module (input). At this time, the data is encrypted before being sent (specific operation).

[0206] Step 3:

[0207] The server uses the database management system to search for surrounding information based on the received location information (input), and extracts information about restaurants, tourist spots, dangerous road conditions, and other information from the database near the current location (output).

[0208] Step 4:

[0209] The server generates the most appropriate information for the user based on the search results (input). For example, if the user is a pedestrian, it will prioritize information about restaurants and weather within walking distance (output).

[0210] Step 5:

[0211] The server sends the generated optimized information to the terminal (input). The sent information includes audio data, text data, and video data (output).

[0212] Step 6:

[0213] The device then provides the received information to the user via voice or an AR screen. For example, it may say, "Here are some recommended restaurants," and display a map on the AR screen (specifically, an example of the area around Shinjuku Station).

[0214] Step 7:

[0215] When a user starts a conversation with someone who speaks a foreign language, the device's microphone captures the conversation (input) and sends the audio data to a server (output).

[0216] Step 8:

[0217] The server translates the received voice data using a translation engine (input). For example, to translate from English to Japanese, it uses the Google Translate API (output).

[0218] Step 9:

[0219] The server generates the translation result and sends it to the terminal (input), which contains the translated text data (output).

[0220] Step 10:

[0221] The device provides the translation results to the user as audio or subtitles, for example, by displaying them as subtitles to enable the user to converse smoothly (specific operation).

[0222] Step 11:

[0223] When the user inputs a prompt sentence, the terminal transmits the text data or voice data to the server (input).

[0224] Step 12:

[0225] The server uses a generative AI model to parse the prompt (input) and generate an appropriate response (output).

[0226] Step 13:

[0227] The server generates a response and sends it to the terminal (input), which may contain, for example, audio or text data (output).

[0228] Step 14:

[0229] The device then provides the generated response to the user via voice or text. For example, in response to the prompt, "Please tell me some recommended restaurants near Shinjuku Station," the device will provide information about recommended restaurants via voice (specific action).

[0230] (Application example 1)

[0231] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0232] In logistics centers, workers often face communication barriers when they need to efficiently receive work instructions over a wide area or when they need to use a foreign language. In addition, there is a need for real-time detection and response to work instructions and dangerous situations within large-scale facilities. These challenges must be addressed.

[0233] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0234] In this invention, the server includes a means for acquiring location information, a means for searching for and generating information about the surrounding area based on the acquired location information, a means for displaying the generated information, and a means for acquiring and displaying work instructions. This enables workers at a logistics center to efficiently receive instructions and perform appropriate work. Furthermore, a system including a means for capturing audio in different languages, a means for translating the captured audio into a specified language, a means for providing the translated results, and a means for displaying the translated results as support information facilitates communication between workers. Furthermore, a system including a means for capturing video of the surrounding area using a 360-degree camera, a means for analyzing the captured video to detect dangerous situations, a means for alerting users of detected danger information, and a means for improving work efficiency at logistics facilities enables improved safety and optimized work efficiency.

[0235] A "means for acquiring location information" is a device, equipment, or software that uses a GPS sensor or other location detection technology to identify and collect information about a person's current location.

[0236] "Means for searching and generating surrounding information" refers to a program or device that searches a database for information about the relevant area or location based on the acquired location information and generates the necessary information.

[0237] "Means for displaying generated information" refers to devices such as displays and AR / VR equipment that visually provide workers or users with the collected and generated information, or software for such devices.

[0238] "Means for obtaining and displaying work instructions" refers to devices or software that include a display or visual presentation method for obtaining specific work instructions from a server or other information source and displaying them in an easy-to-understand manner to the user.

[0239] The "means for capturing audio in different languages" is a microphone and audio collection software for collecting audio data in the languages ​​spoken by the user.

[0240] The "means for translating the captured voice into a specified language" is a translation engine or translation program for converting the captured voice data into another language.

[0241] "Means for providing translated results" means a display or audio player, or software for presenting the translated text or audio to the user.

[0242] The "means for displaying the translation results as support information" refers to a visual presentation device or interface for providing the translated text or audio to the user as work instructions or support information.

[0243] "Means for capturing surrounding images using a 360-degree camera" refers to a 360-degree camera for collecting surrounding images in all directions or an image collection device for such a purpose.

[0244] The "means for analyzing captured video to detect dangerous situations" refers to image analysis software or a program for analyzing acquired video data and detecting dangerous situations or abnormalities.

[0245] "Means for alerting the user to detected danger information" refers to a warning display device or audio alert device, or software for such devices, that notifies the user of detected danger or abnormality.

[0246] "Means for improving work efficiency at logistics facilities" refers to devices and software that provide users with appropriate work instructions and support information to make work within logistics facilities more efficient and smooth the flow of work.

[0247] MODE FOR CARRYING OUT THE INVENTION

[0248] A specific embodiment of the present invention will be described below. This embodiment is a system that uses smart glasses to improve work efficiency in a logistics center.

[0249] Overall system overview

[0250] The system consists of the following main components:

[0251] 1. Smart glasses (terminal)

[0252] 2. Server (backend system)

[0253] 3. Network Communication Infrastructure

[0254] Smart Glasses Configuration

[0255] The smart glasses have the following features:

[0256] GPS sensor: A sensor for obtaining location information.

[0257] 360-degree camera: A camera for capturing images of the surroundings.

[0258] Microphone: A device for capturing sound.

[0259] Speaker: A device for outputting sound.

[0260] Display: A display that can be displayed in AR / VR mode.

[0261] Network communication module: A module for communicating with the server.

[0262] Server Configuration

[0263] The server has the following features:

[0264] Database management system: Manages location-based surrounding information and work instructions.

[0265] Translation engine: Translates audio between different languages.

[0266] Video analysis engine: Analyzes video captured by a 360-degree camera to detect dangerous situations.

[0267] Information services: Providing necessary information such as weather forecasts and logistics information.

[0268] Information acquisition and provision process

[0269] 1. Obtaining location information

[0270] The GPS sensor in the smart glasses captures current location information and sends it to a server.

[0271] 2. Information retrieval and generation

[0272] Based on the received location information, the server searches the database for work instructions within the logistics center and generates optimized information.

[0273] 3. Displaying Information

[0274] Work instructions and surrounding information are displayed in AR on the smart glasses display and provided to workers, while information is also provided via audio.

[0275] 4. Voice translation function

[0276] When a worker initiates a conversation with a colleague who speaks a foreign language, the smart glasses' microphone captures the audio and sends it to the server, where the translation engine translates the speech and sends the results back to the smart glasses, where the translation results are displayed as subtitles on the display.

[0277] 5. Detecting dangerous situations

[0278] The 360-degree camera captures the surroundings in real time and sends it to a server. The server uses a video analytics engine to detect dangerous situations and sends that information to the smart glasses, which then provide a warning via AR display and audio.

[0279] Specific examples

[0280] For example, if a user working in a logistics center wants to know the location of a specific shelf, the smart glasses will obtain the location information and send it to the server. The server will then retrieve the location information from the database and display it on the user's glasses in AR. Furthermore, when the same user communicates with a worker who speaks a foreign language, the glasses can capture their voice and display the translation results on the display, enabling smooth conversation.

[0281] Example prompts to input to a generative AI model:

[0282] "You work at a logistics center in Tokyo. Please use the information below to navigate to the appropriate shelves and communicate with the foreign workers. Your current location is 35.6895, 139.6917."

[0283] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0284] Step 1:

[0285] The device's GPS sensor acquires the current location information. The input is location information, and the output is the acquired location data. The location data acquired from the GPS sensor is sent to the server.

[0286] Step 2:

[0287] The server analyzes the received location data and searches the database for work instructions within the logistics center based on that location information. The input is location data, and the output is the corresponding work instruction data. The database management system is used to generate the optimal work instructions.

[0288] Step 3:

[0289] The server sends the generated work instructions to the terminal. The terminal's display provides the work instructions to the user in an AR display. The input is the work instruction data, and the output is the AR display and audio instructions. The display and speaker are used to provide information visually and audibly.

[0290] Step 4:

[0291] The device's microphone captures audio in different languages. The input is audio data, and the output is the captured audio file. The audio data is sent to the server.

[0292] Step 5:

[0293] The server's translation engine analyzes the captured voice data and translates it into the specified language. The input is the voice data, and the output is the translated text data. The translation engine is used to generate an accurate translation.

[0294] Step 6:

[0295] The server sends the translation results to the terminal, and the terminal's display displays the translation results as subtitles. The input is the translated text data, and the output is the subtitle display. Visual information is provided to the user through the display.

[0296] Step 7:

[0297] The device's 360-degree camera captures the surroundings. The input is the video data, and the output is the captured video file. The video data is sent to the server.

[0298] Step 8:

[0299] The server's video analysis engine analyzes the captured video data and detects dangerous situations. The input is the video data and the output is the detected danger information. The video analysis engine is used to identify dangerous situations.

[0300] Step 9:

[0301] The server sends the detected danger information to the terminal, and the terminal's display and speaker provide a warning to the user. The input is the detected danger information, and the output is a warning display and audio warning. The warning is conveyed to the user using the display and speaker.

[0302] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0303] The smart ARVR glasses system of the present invention is a multifunctional device that allows users to obtain various information in real time, enabling efficient communication and work support. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusting the information provided according to the user's emotions, it can provide a more personalized user experience.

[0304] System configuration

[0305] The system consists of the following main components:

[0306] 1. Device (Smart ARVR Glasses)

[0307] GPS sensor

[0308] 360-degree camera

[0309] microphone

[0310] speaker

[0311] Display (AR / VR mode switchable)

[0312] Network Communication Module

[0313] Emotion Engine

[0314] 2. Server

[0315] Database Management Systems

[0316] Translation Engine

[0317] Video Analysis Engine

[0318] Sentiment Analysis Engine

[0319] Various information services such as weather forecasts, restaurant information, and tourist information

[0320] Information acquisition and provision process

[0321] When a user puts on the smart ARVR glasses and begins using them, the device first obtains its current location using the GPS sensor. This location information is then sent to the server via the network communication module. The server then uses the received location information to search its database for information on nearby restaurants, tourist spots, dangerous roads, weather forecasts, and other information, generating optimized information. The generated information is then sent to the device and provided to the user via voice or an AR screen.

[0322] Emotion Engine Functions

[0323] The emotion engine recognizes the user's current emotion from their voice, facial expressions, and actions, and uses the results to adjust the behavior of the entire system. For example, if the user is feeling stressed, the system will adjust to provide more concise and friendly information.

[0324] Machine translation function

[0325] When a user starts a conversation with someone speaking a foreign language, the device's microphone captures the voice, and the captured voice data is sent to the server along with emotional information. The server then passes the voice data and emotional information through a translation engine to translate the speech according to the user's emotions. The translation results are then sent to the device and provided to the user as audio or subtitles.

[0326] Check road conditions in real time and avoid accidents

[0327] While the user is moving, the device's 360-degree camera captures images of the surroundings in real time. This video data, along with emotional information, is sent to a server, which then uses a video analysis engine to detect dangerous situations and potential accidents. The detection results are then sent to the device, and a warning is displayed on the AR screen or via audio based on the user's emotional state.

[0328] Work support

[0329] When a user arrives at a designated work area at a manufacturing or logistics site, the device transmits location information and emotion information to a server. The server searches a database for relevant work procedures based on the location and emotion information, and transmits the retrieved work procedures to the device. The device then displays the procedures in AR, adjusted according to the user's emotion.

[0330] Customer service support

[0331] When a user starts a conversation with a customer while serving them, the device uses its voice recognition function to capture the conversation. The captured conversation data is sent to a server along with emotional information, and an analysis engine generates an optimal response example. The generated response example is sent to the device and provided to the user as audio or subtitles.

[0332] English conversation lessons

[0333] When a user starts an English conversation lesson, the device switches to VR mode and displays a generated AI model avatar. The user converses with the avatar, and their utterances and emotional information are captured by the device and sent to the server. The server analyzes the utterances and emotional information, generates appropriate responses, and sends them to the device. The device then provides the user with responses tailored to their emotions through the avatar.

[0334] Specific examples

[0335] When a user is walking near Shinjuku Station, the device acquires location information and sends it to the server. The server searches for recommended restaurant information and weather information around Shinjuku Station, generates optimal information based on the location information, and sends it to the device. The device then announces, "Here are some recommended restaurants," and displays the information on the AR screen. If the user shows signs of fatigue, the system also suggests nearby rest areas and coffee shops.

[0336] In addition, if a user is conversing in English with a foreign tourist, the device captures the conversation and sends it to the server. The server translates the English into Japanese and sends the translation result to the device. The device then displays the translated Japanese as subtitles, allowing the user to communicate smoothly. Furthermore, if the user is feeling tense, the system provides translation results using shorter sentences and simpler phrases.

[0337] As described above, the smart ARVR glasses system of the present invention integrates multiple functions, such as location information, voice capture, translation, video analysis, and emotion recognition, to provide users with a wide range of information support in real time. Furthermore, by incorporating an emotion engine, it is possible to flexibly respond to the user's emotional state, realizing a personalized user experience.

[0338] The processing flow will be explained below.

[0339] Information acquisition and provision process

[0340] Step 1:

[0341] Device: Uses the GPS sensor to obtain current location information.

[0342] Step 2:

[0343] Terminal: Sends the acquired location information to the server via the network communication module.

[0344] Step 3:

[0345] Server: Based on the received location information, the server searches a database for information such as nearby restaurants, tourist spots, dangerous roads, and weather forecasts.

[0346] Step 4:

[0347] Server: Optimizes search results and generates the information provided.

[0348] Step 5:

[0349] Server: Sends the generated information to the terminal.

[0350] Step 6:

[0351] Terminal: Provides received information to the user via voice or an AR screen.

[0352] Emotion Engine Functions

[0353] Step 1:

[0354] Device: Uses a microphone and camera to capture the user's voice and facial expressions.

[0355] Step 2:

[0356] Terminal: Sends captured audio and video data to the server.

[0357] Step 3:

[0358] Server: Recognizes user emotions using an emotion analysis engine.

[0359] Step 4:

[0360] Server: Adjust how information is presented based on perceived emotions.

[0361] Machine translation function

[0362] Step 1:

[0363] User: Start a conversation with someone who speaks a foreign language.

[0364] Step 2:

[0365] Device: Uses the microphone to capture audio.

[0366] Step 3:

[0367] Device: Sends captured audio data to the server.

[0368] Step 4:

[0369] Server: The voice data is passed through a translation engine and translated into the specified language.

[0370] Step 5:

[0371] Server: Adjusts translation results taking into account the user's emotional information.

[0372] Step 6:

[0373] Server: Sends the translation results to the device.

[0374] Step 7:

[0375] Device: Providing translated audio and subtitles to users.

[0376] Check road conditions in real time and avoid accidents

[0377] Step 1:

[0378] Device: Uses a 360-degree camera to capture footage of the surroundings in real time.

[0379] Step 2:

[0380] Terminal: Sends captured video data to the server.

[0381] Step 3:

[0382] Server: Uses a video analytics engine to detect dangerous situations and potential accidents.

[0383] Step 4:

[0384] Server: Adjust the warning content taking into account the user's emotional information.

[0385] Step 5:

[0386] Server: Sends the detection results to the device.

[0387] Step 6:

[0388] Device: Provides warning information to users via voice or AR screen.

[0389] Work support

[0390] Step 1:

[0391] User: Arrives at a designated work area on a manufacturing or logistics site.

[0392] Step 2:

[0393] Device: Obtains current location information and sends it to the server.

[0394] Step 3:

[0395] Server: Based on the location information, it searches the database for work procedures relevant to that area.

[0396] Step 4:

[0397] Server: Recognizes user emotions using an emotion analysis engine.

[0398] Step 5:

[0399] Server: Adjusts work procedures based on perceived emotions.

[0400] Step 6:

[0401] Server: Sends the adjusted work instructions to the device.

[0402] Step 7:

[0403] Terminal: Work procedures are displayed in AR in the user's field of view.

[0404] Customer service support

[0405] Step 1:

[0406] User: Starts a conversation with a customer.

[0407] Step 2:

[0408] Device: Use voice recognition to capture what you say.

[0409] Step 3:

[0410] Terminal: Sends captured conversation data to the server.

[0411] Step 4:

[0412] Server: Analyzes conversation data and generates optimal response examples.

[0413] Step 5:

[0414] Server: Recognizes user emotions using an emotion analysis engine.

[0415] Step 6:

[0416] Server: Adjusts response examples based on the recognized emotion.

[0417] Step 7:

[0418] Server: Sends the adjusted response examples to the device.

[0419] Step 8:

[0420] Device: Provide users with examples of what works with audio and subtitles.

[0421] English conversation lessons

[0422] Step 1:

[0423] User: Switches to VR mode to begin an English conversation lesson.

[0424] Step 2:

[0425] Terminal: Display the generated AI model avatar.

[0426] Step 3:

[0427] User: Converse with an AI avatar.

[0428] Step 4:

[0429] Device: Captures user speech and emotion information and sends it to the server.

[0430] Step 5:

[0431] Server: Analyzes the user's comments and emotional information and generates an appropriate response.

[0432] Step 6:

[0433] Server: Sends the adjusted response to the device.

[0434] Step 7:

[0435] Terminal: Provides tailored responses to users through avatars.

[0436] Example 2

[0437] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0438] Conventional information provision systems can only provide limited information based on the user's location information, making it difficult to respond flexibly to the user's emotions and situation. Furthermore, communication support in different languages ​​and detection and warning of dangerous situations can only respond uniformly without considering the user's emotions, so there is a need to improve the user experience.

[0439] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0440] In this invention, the server includes a means for acquiring location information, a means for searching and generating surrounding information based on the acquired location information, a means for displaying the generated information, and a means for recognizing the user's emotions and adjusting the information according to the emotions. This makes it possible to not only provide surrounding information based on the user's location information, but also to provide appropriate information, translation, and warnings according to the user's emotions.

[0441] "Location Information" means data of a user's current physical geographic coordinates obtained using a GPS sensor or other location detection device.

[0442] "Nearby information" refers to information related to the user's current location, such as nearby restaurants, tourist attractions, and weather information.

[0443] "Means for recognizing emotions" refers to technology that analyzes the user's voice, facial expressions, and movement data to estimate the user's emotional state.

[0444] A "360-degree camera" is a camera device capable of shooting in all directions, capturing a wide range of images at once.

[0445] A "dangerous situation" is a situation in the surroundings that may affect the user's safety, such as an accident or an obstacle.

[0446] "Adjusting according to emotions" means appropriately changing the information and warning content provided based on the user's current emotional state.

[0447] "Speech in a different language" refers to speech data in a language other than the language spoken by the user.

[0448] "Means for translating" refers to technology for converting audio data provided in a particular language into another language.

[0449] "Generated information" refers to data such as restaurant information and warnings generated based on location information, surrounding conditions, and the user's emotional state.

[0450] "Means of providing to the user" refers to the technology or device used to convey the generated information to the user in the form of audio, AR screen, text, etc.

[0451] The smart ARVR glasses system of the present invention is a multifunctional device that allows users to obtain various information in real time, enabling efficient communication and work support. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusting the information provided according to the user's emotions, it can provide a more personalized user experience.

[0452] System configuration

[0453] The system consists of the following main components:

[0454] 1. Device (Smart ARVR Glasses)

[0455] GPS sensor

[0456] 360-degree camera

[0457] microphone

[0458] speaker

[0459] Display (AR / VR mode switchable)

[0460] Network Communication Module

[0461] Emotion Engine

[0462] 2. Server

[0463] Database Management Systems

[0464] Translation Engine

[0465] Video Analysis Engine

[0466] Sentiment Analysis Engine

[0467] Various information services such as weather forecasts, restaurant information, and tourist information

[0468] Information acquisition and provision process

[0469] When a user puts on the smart ARVR glasses and begins using them, the device first obtains its current location using the GPS sensor. This location information is then sent to the server via the network communication module. The server then uses the received location information to search a database for information such as nearby restaurants, tourist spots, road hazards, and weather forecasts, and generates optimized information. The generated information is then sent to the device and provided to the user via voice or an AR screen.

[0470] Emotion Engine Functions

[0471] The emotion engine recognizes the user's current emotion from their voice, facial expressions, and actions, and uses the results to adjust the behavior of the entire system. For example, if the user is feeling stressed, the system will adjust to provide more concise and friendly information.

[0472] Machine translation function

[0473] When a user starts a conversation with someone speaking a foreign language, the device's microphone captures the voice, and the captured voice data is sent to the server along with emotional information. The server then passes the voice data and emotional information through a translation engine to translate the speech according to the user's emotions. The translation results are then sent to the device and provided to the user as audio or subtitles.

[0474] Check road conditions in real time and avoid accidents

[0475] While the user is moving, the device's 360-degree camera captures images of the surroundings in real time. This video data, along with emotional information, is sent to a server, which then uses a video analysis engine to detect dangerous situations and potential accidents. The detection results are then sent to the device, and a warning is displayed on the AR screen or via audio based on the user's emotional state.

[0476] Work support

[0477] When a user arrives at a designated work area at a manufacturing or logistics site, the device transmits location information and emotion information to a server. The server searches a database for relevant work procedures based on the location and emotion information, and transmits the retrieved work procedures to the device. The device then displays the procedures in AR, adjusted according to the user's emotion.

[0478] Customer service support

[0479] When a user starts a conversation with a customer while serving them, the device uses its voice recognition function to capture the conversation. The captured conversation data is sent to a server along with emotional information, and an analysis engine generates an optimal response example. The generated response example is sent to the device and provided to the user as audio or subtitles.

[0480] English conversation lessons

[0481] When a user starts an English conversation lesson, the device switches to VR mode and displays a generated AI model avatar. The user converses with the avatar, and their utterances and emotional information are captured by the device and sent to the server. The server analyzes the utterances and emotional information, generates appropriate responses, and sends them to the device. The device then provides the user with responses tailored to their emotions through the avatar.

[0482] Specific examples

[0483] For example, if a user is walking near Shinjuku Station, the device acquires location information and sends it to the server. The server searches for recommended restaurant information and weather information around Shinjuku Station, generates optimal information based on the location information, and sends it to the device. The device then announces, "Here are some recommended restaurants," and displays the information on the AR screen. If the user shows signs of fatigue, the system will also suggest nearby rest areas and coffee shops.

[0484] In addition, if a user is conversing in English with a foreign tourist, the device captures the conversation and sends it to the server. The server translates the English into Japanese and sends the translation result to the device. The device then displays the translated Japanese as subtitles, allowing the user to communicate smoothly. Furthermore, if the user is feeling tense, the system provides translation results using shorter sentences and simpler phrases.

[0485] Prompt Sentence Examples

[0486] An example prompt using a generative AI model would be:

[0487] "Please explain the process of providing information when a user walking near Shinjuku Station wants to check nearby restaurant recommendations. Also, please provide a specific example of how to respond if the user feels fatigued."

[0488] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0489] Step 1:

[0490] Acquiring and sending location information

[0491] The device uses the GPS sensor to obtain its current location.

[0492] Input: The user's current location.

[0493] Output: Real-time location data.

[0494] The acquired location information is transmitted to a server via a network communication module.

[0495] Operation: The device will announce "Current location obtained" and send data such as "Location information: Near Shinjuku Station" to the server.

[0496] Step 2:

[0497] Information Retrieval and Optimization

[0498] Based on the received location information, the server searches its database for nearby restaurants, tourist attractions, weather information, dangerous road information, and more.

[0499] Input: Location information sent from the device.

[0500] Output: Neighborhood information data as search results.

[0501] The server optimizes the acquired information and generates information to be presented to the user.

[0502] What it does: The server sends a query to the database to search for "recommended restaurants around Shinjuku Station." Based on the search results, it logs "Restaurant information generated."

[0503] Step 3:

[0504] Information provision and user emotional response

[0505] The device receives the generated information and provides it to the user via AR screen and audio.

[0506] Input: Optimized information data sent from the server.

[0507] Output: Visual and audio information displayed and notified to the user.

[0508] The device captures the user's reactions and emotions to the information provided.

[0509] How it works: The device will say "Here's a recommended restaurant" and display the restaurant's location on the AR screen. The device will capture the user's reaction and store it as emotion information.

[0510] Step 4:

[0511] Emotion Engine Operation

[0512] The server receives the user's voice, facial expression, and movement data and analyzes it using an emotion analysis engine.

[0513] Input: User voice, facial expression, and movement data sent from the device.

[0514] Output: Sentiment analysis result data.

[0515] The next information provided is adjusted according to the user's emotions.

[0516] Operation: If data is captured showing that the user is feeling tired, the emotion analysis engine detects "fatigue" and executes a process to "suggest nearby rest areas." Related information is sent to the device.

[0517] Step 5:

[0518] Machine translation function

[0519] The device uses a microphone to capture the user's voice and transmits it to the server along with emotional information.

[0520] Input: User's voice data and emotion information.

[0521] Output: Captured audio data and emotion information.

[0522] The server uses a translation engine to translate the captured audio.

[0523] Input: Voice data and emotion information sent from the device.

[0524] Output: Translation result data.

[0525] The system adjusts the translation based on the user's emotions and sends the translation results to the device.

[0526] Operation: The device notifies the user that "Audio data has been sent for translation," and the server logs "English to Japanese translation completed," and sends the translation results to the device.

[0527] Step 6:

[0528] Real-time road conditions and alerts

[0529] The device uses a 360-degree camera to capture images of the surroundings and send them to a server.

[0530] Input: Captured ambient video data and emotion information.

[0531] Output: Real-time video data.

[0532] The server uses a video analytics engine to detect dangerous situations.

[0533] Input: Video data sent from the device.

[0534] Output: Hazardous situation detection data.

[0535] The detection results are sent to the terminal and a warning is issued to the user.

[0536] What it does: The device displays "Capturing real-time video," the server logs "Dangerous intersection detected," and sends a warning to the device saying "Be careful."

[0537] Step 7:

[0538] Work support

[0539] When the user arrives at the work area, the device transmits location information and emotion information to the server.

[0540] Input: Location information obtained in the working area and user emotion information.

[0541] Output: Sending data to the server.

[0542] The server searches a database for relevant work procedures based on the location information and emotion information.

[0543] Input: Location and emotion information sent from the device.

[0544] Output: The searched work procedure data.

[0545] The acquired work procedures are sent to the terminal, and procedures adjusted according to the user's emotions are displayed in AR in the user's field of view.

[0546] Operation: The device issues a voice notification saying "Arrived at work area" and sends its location information to the server. The server logs "Work procedure generated" and sends it to the device.

[0547] Step 8:

[0548] Customer service support

[0549] The device uses voice recognition to capture conversations with customers.

[0550] Input: Customer conversation audio data.

[0551] Output: Captured conversation data and emotion information.

[0552] The server uses an analysis engine to generate the best possible response.

[0553] Input: Conversational data and emotional information sent from the device.

[0554] Output: Generated example data.

[0555] The generated examples are sent to the terminal and provided to the user as audio and subtitles.

[0556] How it works: The device displays "Capturing conversation," the server logs "Suggested response generated," and sends a suggestion to the device saying, "Here's what to say next."

[0557] Step 9:

[0558] English conversation lessons

[0559] The device switches to VR mode and displays the generated AI model avatar. The user's speech and emotional information are sent to the server.

[0560] Input: User utterance data and emotion information.

[0561] Output: Captured speech data and emotion information.

[0562] The server analyzes the content of the statement and emotional information and generates an appropriate response.

[0563] Input: Speech data and emotion information sent from the device.

[0564] output: The generated response data.

[0565] The generated response is sent to the terminal and provided to the user through the avatar.

[0566] Operation: The device announces "Start English conversation lesson" and displays an avatar. The server logs "Analyzing user's comment" and generates an appropriate response and sends it to the device.

[0567] The above are the specific processing steps of the program of this system.

[0568] (Application example 2)

[0569] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0570] Conventional ARVR systems and information acquisition systems provide information uniformly without considering the user's state or emotions, making it difficult to achieve a personalized user experience. While efficient real-time information provision is particularly required in logistics centers and factories, conventional systems have difficulty responding flexibly to the worker's emotions and situation. Furthermore, smooth communication between different languages ​​is difficult because translation results are provided without taking emotions into account. Furthermore, detection of dangerous situations and warnings during work do not reflect emotional information, making it difficult to respond appropriately to the user's state.

[0571] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring location information, means for displaying generated information, means for recognizing emotions, means for adjusting information to be provided according to the recognized emotions, means for capturing voices in different languages, means for providing translation results, means for translating the captured voices into a specified language, means for capturing surrounding video using a 360-degree camera, means for analyzing the captured video to detect dangerous situations, means for alerting the user of detected danger information, and means for acquiring emotional information and adjusting the content of the alert based on the analysis results. This makes it possible to provide personalized information based on the user's location information and emotional state and improve safety.

[0572] 1. "Means for obtaining location information" refers to a device that uses a GPS sensor or other location-determining technology to determine the user's current location.

[0573] 2. "Means for searching and generating surrounding information" refers to a processing system that searches a database for information about the surrounding environment and available services based on location information and provides it to the user.

[0574] 3. "Means for displaying generated information" refers to devices such as displays and speakers that provide acquired information to the user visually or audibly.

[0575] 4. "Means for recognizing emotions" refers to software and sensors for detecting and analyzing the user's emotional state from their voice, facial expressions, movements, etc.

[0576] 5. "Means for adjusting the information provided in response to the recognized emotion" means a processing system that changes the content or format of the information provided based on the detected emotion information.

[0577] 6. "Means for capturing audio in different languages" means microphones and speech analysis devices for collecting audio data corresponding to multiple languages.

[0578] 7. "Means for translating captured audio into a specified language" means translation software or services that convert captured audio data into another language.

[0579] 8. "Means for providing translation results" means a display or speaker for providing the translated audio or text to the user.

[0580] 9. "Means for capturing surrounding images using a 360-degree camera" means a camera device for capturing images in all directions.

[0581] 10. "Means for analyzing captured video to detect dangerous situations" means software that analyzes captured video data and recognizes dangerous situations and obstacles.

[0582] 11. "Means for warning the user of detected danger information" refers to a device that issues a visual or audible warning to the user based on the analysis results.

[0583] 12. "Means for acquiring emotional information and adjusting the content of warnings based on the analysis results" refers to a processing system for changing the content and format of warnings based on the user's emotional state.

[0584] Overall system overview

[0585] The system of this invention uses smart ARVR glasses to provide information and support to logistics center workers. This system acquires location information, analyzes emotional information, and provides the generated information to users, enabling efficient and safe work.

[0586] Hardware Configuration

[0587] 1. Device (Smart ARVR Glasses):

[0588] GPS sensor: Obtain location information

[0589] 360-degree camera: Capture footage of your surroundings

[0590] Microphone, speaker: Audio capture and output

[0591] Display (AR / VR mode switchable)

[0592] Network communication module: Data communication with the server

[0593] Emotion engine: Recognizes user emotions and adjusts information delivery accordingly

[0594] 2. Server:

[0595] Database Management System (DBMS): MariaDB

[0596] Translation engine: Google Cloud Translation API

[0597] Video analysis engine: OpenCV

[0598] Sentiment analysis engine: IBM Watson(R) Tone Analyzer

[0599] Various information service APIs (work procedures, hazard information, etc.)

[0600] Software Configuration and Processing

[0601] The server processes information using the following means.

[0602] 1. Obtaining and sending location information:

[0603] The device's GPS sensor acquires location information and transmits it to the server via the network communication module. The server then references the database based on the location information to search for and generate relevant work procedures.

[0604] 2. Acquisition and analysis of emotional information:

[0605] The emotion engine captures the user's voice and facial expressions and sends them to the emotion analysis engine. The server analyzes the emotional information and generates optimal information according to the user's state.

[0606] 3. Video capture and analysis:

[0607] The 360-degree camera captures the surroundings and sends the video data to a server, which uses a video analytics engine to detect dangerous situations and alert the user.

[0608] Specific examples

[0609] 1. Display of work procedure:

[0610] When the user arrives at the designated work area, the device acquires the user's location information and sends it to the server. The server searches the database for relevant work procedures based on the location information, and sends the acquired work procedures to the device. The device then displays the procedures in AR, adjusted according to the user's emotions.

[0611] Example prompt: Tell me the procedure, and the location is "34.052235, -118.243683"

[0612] 2. Notification of dangerous situations:

[0613] While the user is moving, a 360-degree camera captures the surroundings and sends it to a server. The server uses a video analysis engine to detect dangerous situations and warn the user. Emotional information is also analyzed, and the warning content is adjusted accordingly.

[0614] Example prompt: Tells about danger around you and makes you feel nervous

[0615] 3. Multilingual Translation:

[0616] When a user starts a conversation with someone who speaks a different language, the device's microphone captures the voice and sends it along with emotional information to the server. The server then uses a translation engine to translate the voice into the specified language and sends the translation result to the device. The translation result is adjusted appropriately according to the user's emotional state.

[0617] Example prompt: Translate the conversation and the emotion is "stress"

[0618] As a result, the present invention enables workers in a logistics center to receive personalized support and perform their work safely and efficiently.

[0619] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0620] Program processing steps

[0621] Process flow for displaying work procedures

[0622] Step 1:

[0623] The device's GPS sensor acquires the user's current location. The input is location data from the GPS sensor, and the output is the acquired location information. Specifically, the GPS sensor periodically acquires location data and stores it in internal memory.

[0624] Step 2:

[0625] The location information acquired by the terminal is sent to the server via the network communication module. The input is location information, and the output is location data sent to the server. Specifically, the terminal converts the location information into packets and sends them to the server via the network communication module over the Internet.

[0626] Step 3:

[0627] The server refers to the database management system (DBMS) based on the location information it receives and searches for related work procedures. The input is the location information received by the server, and the output is the searched work procedure data. Specifically, the server generates an SQL query and retrieves related work procedures from the database.

[0628] Step 4:

[0629] The server sends the retrieved work procedure to the terminal via the network communication module. The input is the retrieved work procedure data, and the output is the work procedure to be sent to the terminal. Specifically, the server converts the work procedure data into packets and sends them to the terminal via the network communication module.

[0630] Step 5:

[0631] The device's emotion engine analyzes the user's voice and facial expressions to obtain emotional information. The input is voice data and facial expression data, and the output is emotional information. Specifically, the device captures voice and facial expressions using a microphone and camera, and processes them using the built-in emotion analysis engine.

[0632] Step 6:

[0633] The device adjusts the displayed work steps based on the emotional information it acquires. The input is emotional information and work step data, and the output is the adjusted work steps. Specifically, the emotion engine analyzes the emotional information and changes the display format and content to suit the user's state.

[0634] Step 7:

[0635] The device displays the adjusted work procedure in AR in the user's field of view. The input is the adjusted work procedure data, and the output is visual information provided to the user. Specifically, the device display displays the adjusted work procedure in AR mode, allowing the user to visually confirm it.

[0636] Processing flow for notification of dangerous situations

[0637] Step 1:

[0638] The device's 360-degree camera captures the surroundings. The input is the camera's video data, and the output is the captured video data. Specifically, the camera periodically captures video and stores it in its internal memory.

[0639] Step 2:

[0640] The video data captured by the terminal is sent to the server via the network communication module. The input is video data, and the output is video data sent to the server. Specifically, the terminal converts the video data into packets and sends them to the server via the network.

[0641] Step 3:

[0642] The video data received by the server is analyzed by a video analysis engine to detect dangerous situations. The input is video data and the output is detected danger information. Specifically, the video analysis engine analyzes the video data and automatically detects dangerous situations.

[0643] Step 4:

[0644] The server sends danger information to the terminal via the network communication module. The input is danger information, and the output is danger data sent to the terminal. Specifically, the server converts the danger information into packets and sends them to the terminal via the network communication module.

[0645] Step 5:

[0646] The device's emotion engine analyzes the user's voice and facial expressions to obtain emotional information. The input is voice data and facial expression data, and the output is emotional information. Specifically, the device captures voice and facial expressions using a microphone and camera, and processes them using the built-in emotion analysis engine.

[0647] Step 6:

[0648] The device adjusts the warning content based on danger information and emotional information. The input is danger information and emotional information, and the output is the adjusted warning content. Specifically, the emotion engine analyzes the emotional information and changes the content and format of the warning to suit the user's state.

[0649] Step 7:

[0650] The device notifies the user of the adjusted warning visually or audibly. The input is the adjusted warning content, and the output is the warning information provided to the user. Specifically, the device's display or speaker displays or notifies the user of the adjusted warning, allowing the user to respond immediately.

[0651] The above are the specific processing steps for carrying out the invention.

[0652] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0653] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0654] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0655] [Second embodiment]

[0656] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0657] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0658] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0659] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0660] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0661] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0662] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0663] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0664] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0665] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0666] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0667] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0668] The smart ARVR glasses system of the present invention is a multi-functional device that allows users to obtain various information in real time and provide efficient communication and work assistance.

[0669] System configuration

[0670] The system consists of the following main components:

[0671] 1. Device (Smart ARVR Glasses)

[0672] GPS sensor

[0673] 360-degree camera

[0674] microphone

[0675] speaker

[0676] Display (AR / VR mode switchable)

[0677] Network Communication Module

[0678] 2. Server

[0679] Database Management Systems

[0680] Translation Engine

[0681] Video Analysis Engine

[0682] Various information services such as weather forecasts, restaurant information, and tourist information

[0683] Information acquisition and provision process

[0684] When a user puts on the smart ARVR glasses and begins using them, the device first obtains its current location using the GPS sensor. This location information is then sent to the server via the network communication module. The server then uses the received location information to search its database for information on nearby restaurants, tourist spots, dangerous roads, weather forecasts, and other information, generating optimized information. The generated information is then sent to the device and provided to the user via voice or an AR screen.

[0685] Machine translation function

[0686] When a user starts a conversation with someone who speaks a foreign language, the device's microphone captures the voice and sends the captured voice data to the server. The server then passes the voice data through a translation engine to translate it into the required language. The translation result is then sent to the device and provided to the user as audio or subtitles.

[0687] Check road conditions in real time and avoid accidents

[0688] While the user is moving, the device's 360-degree camera captures video of the surroundings in real time. This video data is sent to a server, which uses a video analytics engine to detect dangerous situations or potential accidents. The results are then sent to the device, and a warning is displayed to the user via audio and an AR screen.

[0689] Work support

[0690] When a user arrives at a designated work area at a manufacturing or logistics site, the device sends location information to a server. The server searches a database for work procedures related to that area based on the location information, and sends the retrieved work procedures to the device. The device then displays the procedures in AR within the user's field of view, helping to improve work efficiency.

[0691] Customer service support

[0692] When a user starts a conversation with a customer while serving them, the device uses its voice recognition function to capture the conversation. The captured conversation data is sent to a server, where an analysis engine generates the most appropriate response. The generated response is sent to the device and provided to the user as audio or subtitles.

[0693] English conversation lessons

[0694] When a user starts an English conversation lesson, the device switches to VR mode and displays a generated AI model avatar. The user converses with the avatar, and their utterances are captured by the device and sent to the server. The server analyzes the utterances, generates appropriate responses, and sends them to the device. The device then provides the generated responses to the user through the avatar.

[0695] Specific examples

[0696] When a user is walking near Shinjuku Station, the device acquires location information and sends it to the server. The server searches for recommended restaurants and weather information around Shinjuku Station, generates the most appropriate information, and sends it to the device. The device then announces, "Here are some recommended restaurants," and displays it on the AR screen.

[0697] In addition, if a user is conversing in English with a foreign tourist, the device captures the conversation and sends it to the server. The server translates the English into Japanese and sends the translation result to the device. The device displays the translated Japanese as subtitles, allowing the user to communicate smoothly.

[0698] As described above, the smart ARVR glasses system of the present invention can provide users with a wide range of information support in real time by integrating multiple functions such as location information, voice capture, translation, and video analysis.

[0699] The processing flow will be explained below.

[0700] Information acquisition and provision process

[0701] Step 1:

[0702] Device: Uses the GPS sensor to obtain current location information.

[0703] Step 2:

[0704] Terminal: Sends the acquired location information to the server via the network communication module.

[0705] Step 3:

[0706] Server: Based on the received location information, the server searches a database for information such as nearby restaurants, tourist spots, dangerous roads, and weather forecasts.

[0707] Step 4:

[0708] Server: Optimizes search results and generates the information provided.

[0709] Step 5:

[0710] Server: Sends the generated information to the terminal.

[0711] Step 6:

[0712] Terminal: Provides received information to the user via voice or an AR screen.

[0713] Machine translation function

[0714] Step 1:

[0715] User: Start a conversation with someone who speaks a foreign language.

[0716] Step 2:

[0717] Device: Uses the microphone to capture audio.

[0718] Step 3:

[0719] Device: Sends captured audio data to the server.

[0720] Step 4:

[0721] Server: The received voice data is passed through a translation engine and translated into the specified language.

[0722] Step 5:

[0723] Server: Sends the translation results to the device.

[0724] Step 6:

[0725] Device: Providing translated audio and subtitles to users.

[0726] Check road conditions in real time and avoid accidents

[0727] Step 1:

[0728] Device: Uses a 360-degree camera to capture footage of the surroundings in real time.

[0729] Step 2:

[0730] Terminal: Sends captured video data to the server.

[0731] Step 3:

[0732] Server: Analyzes the received video data and detects dangerous situations and potential accidents.

[0733] Step 4:

[0734] Server: Sends the detection results to the device.

[0735] Step 5:

[0736] Device: Provides warning information to users via voice or AR screen.

[0737] Work support

[0738] Step 1:

[0739] User: Arrives at a designated work area on a manufacturing or logistics site.

[0740] Step 2:

[0741] Device: Obtains current location information and sends it to the server.

[0742] Step 3:

[0743] Server: Based on the location information, it searches the database for work procedures relevant to that area.

[0744] Step 4:

[0745] Server: Sends search results to the device.

[0746] Step 5:

[0747] Terminal: Work procedures are displayed in AR in the user's field of view.

[0748] Customer service support

[0749] Step 1:

[0750] User: Starts a conversation with a customer.

[0751] Step 2:

[0752] Device: Use voice recognition to capture what you say.

[0753] Step 3:

[0754] Terminal: Sends captured conversation data to the server.

[0755] Step 4:

[0756] Server: Analyzes conversation data and generates optimal response examples.

[0757] Step 5:

[0758] Server: Sends the generated response examples to the device.

[0759] Step 6:

[0760] Device: Provide users with examples of what works with audio and subtitles.

[0761] English conversation lessons

[0762] Step 1:

[0763] User: Switches to VR mode to begin an English conversation lesson.

[0764] Step 2:

[0765] Terminal: Display the generated AI model avatar.

[0766] Step 3:

[0767] User: Converse with an AI avatar.

[0768] Step 4:

[0769] Terminal: Captures what the user says and sends it to the server.

[0770] Step 5:

[0771] Server: Analyzes what the user says and generates an appropriate response.

[0772] Step 6:

[0773] Server: Generates a response and sends it to the device.

[0774] Step 7:

[0775] Terminal: The generated response is presented to the user through an avatar.

[0776] Example 1

[0777] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0778] In conventional smart devices and information systems, functions such as location-based information provision, language translation, and hazard detection were implemented separately, forcing users to use multiple devices and applications simultaneously. Furthermore, because the various functions were not linked in real time, it was difficult to efficiently obtain information or provide work support. Therefore, a system that integrated all functions was needed.

[0779] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0780] In this invention, the server includes means for acquiring location information, means for searching for surrounding information based on the acquired location information, means for generating optimal information based on the searched information, means for displaying the generated information, means for capturing audio in different languages, means for translating the captured audio into a specified language, means for providing the translated result, means for capturing video, means for analyzing the captured video to detect a dangerous situation, means for alerting the user of detected danger information, means for inputting a prompt sentence, means for analyzing the input prompt sentence, means for generating an appropriate response based on the analysis result, and means for providing the generated response. This enables a single system to provide multiple functions including information provision, translation, danger detection, and conversation support based on prompts.

[0781] "Means for acquiring location information" refers to a device or system for measuring the current location and acquiring that information as digital data.

[0782] The "means for searching surrounding information" is a system for extracting information about the region and environment from a database based on the acquired location information.

[0783] "Means for generating optimal information" refers to algorithms or programs that evaluate the searched information, convert it into the most useful form for the user, and provide it to the user.

[0784] The "means for displaying information" refers to a device or interface for visually or audibly presenting the generated information to the user.

[0785] A "means for capturing audio in different languages" is a system for using a device such as a microphone to pick up audio spoken in different languages ​​and record it as digital data.

[0786] A "means for translating speech into a specified language" is software or algorithms that analyze the captured speech data and translate it into a target language.

[0787] "Means for providing translated results" refers to a function or device for outputting translated language data to the user in the form of voice or text.

[0788] A "means for capturing video" is a system that uses a device such as a camera to capture video of the surroundings and record it as digital data.

[0789] "Means for analyzing captured video to detect dangerous situations" refers to algorithms or programs that analyze the acquired video data and recognize dangerous situations or abnormal movements from it.

[0790] The "means for warning the user about danger information" refers to means such as sound, visual display, vibration, etc. for notifying the user of the detected danger information.

[0791] A "means for inputting prompt sentences" is an interface that allows the user to freely enter questions or instructions into the system through text input or voice input.

[0792] The "means for analyzing the input prompt sentence" is a natural language processing algorithm for understanding the prompt sentence input by the user and performing appropriate processing according to its content.

[0793] The "means for generating an appropriate response" is software or a model for generating an accurate and appropriate response to the user based on the analyzed prompt sentence.

[0794] The "means for providing a response" refers to a function or device that outputs the generated response to the user in voice or text form and supports the dialogue with the user.

[0795] The system of the present invention is an integrated information provision system that combines a smart device and a server. This makes it possible to provide multiple functions such as location information, translation, road condition confirmation, and prompt sentence analysis in an integrated manner. The specific configuration and operation procedure are described below.

[0796] System configuration

[0797] The system mainly consists of the following components:

[0798] 1. Terminal (smart device such as smart glasses)

[0799] GPS sensor

[0800] Cameras (including 360-degree cameras)

[0801] microphone

[0802] speaker

[0803] Display (AR / VR mode switchable)

[0804] Network communication module (Wi-Fi and mobile data communication)

[0805] 2. Server

[0806] Database Management Systems

[0807] Translation engine (e.g. Google Translate API)

[0808] Video Analysis Engine

[0809] information search engine

[0810] Generative AI Models

[0811] Main functions and processing flow

[0812] Information provision function

[0813] When a user puts on a smart device and starts using it, the device's GPS sensor acquires its current location. The acquired location information is sent to a server via a network communication module. Based on this location information, the server uses a database management system to search for information about the surrounding area and generate the most appropriate information from the search results. This information is then provided to the user in the form of audio, AR display, or other formats.

[0814] Specific examples

[0815] When a user is walking near Shinjuku Station, the device's GPS sensor acquires location information and sends it to the server. The server searches for recommended restaurants and weather information around Shinjuku Station, generates the most appropriate information, and sends it to the device. The device then announces, "Here are some recommended restaurants," and displays it on the AR screen.

[0816] Machine translation function

[0817] When a user starts a conversation with someone speaking a foreign language, the device's microphone captures the audio and sends it to the server. The server then translates the audio data into the required language through a translation engine and sends the translation results to the device. The translation results are then provided to the user as audio or subtitles.

[0818] Specific examples

[0819] When a user is conversing in English with a foreign tourist, the device's microphone captures the conversation and sends it to the server. The server then passes the voice data through a translation engine to translate the English into Japanese and sends the result to the device. The device then displays the translated Japanese as subtitles, allowing the user to communicate smoothly.

[0820] Road condition check function

[0821] While the user is moving, the device's 360-degree camera captures video of the surroundings. This video data is sent to a server, which uses a video analysis engine to detect dangerous situations or potential accidents. The results are then sent to the device, where they are used to warn the user via audio and AR displays.

[0822] Prompt sentence analysis function

[0823] When a user enters a prompt, the device sends the text or voice data to the server, which uses a generative AI model to analyze the prompt and generate an appropriate response, which is then sent to the device and provided to the user as voice or text.

[0824] Prompt Sentence Examples

[0825] "Please tell me some recommended restaurants near Shinjuku Station."

[0826] As described above, the system of the present invention provides multiple functions, including location information, voice capture, translation, video analysis, and prompt sentence analysis, in an integrated manner, and can provide multifunctional information support to users in real time.

[0827] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0828] Step 1:

[0829] The user wears a smart device and starts the system. The device acquires the current location information using the GPS sensor. This location information (input) is stored on the device as coordinate data (output).

[0830] Step 2:

[0831] The device sends the acquired location information to the server via the network communication module (input). At this time, the data is encrypted before being sent (specific operation).

[0832] Step 3:

[0833] The server uses the database management system to search for surrounding information based on the received location information (input), and extracts information about restaurants, tourist spots, dangerous road conditions, and other information from the database near the current location (output).

[0834] Step 4:

[0835] The server generates the most appropriate information for the user based on the search results (input). For example, if the user is a pedestrian, it will prioritize information about restaurants and weather within walking distance (output).

[0836] Step 5:

[0837] The server sends the generated optimized information to the terminal (input). The sent information includes audio data, text data, and video data (output).

[0838] Step 6:

[0839] The device then provides the received information to the user via voice or an AR screen. For example, it may say, "Here are some recommended restaurants," and display a map on the AR screen (specifically, an example of the area around Shinjuku Station).

[0840] Step 7:

[0841] When a user starts a conversation with someone who speaks a foreign language, the device's microphone captures the conversation (input) and sends the audio data to a server (output).

[0842] Step 8:

[0843] The server translates the received voice data using a translation engine (input). For example, to translate from English to Japanese, it uses the Google Translate API (output).

[0844] Step 9:

[0845] The server generates the translation result and sends it to the terminal (input), which contains the translated text data (output).

[0846] Step 10:

[0847] The device provides the translation results to the user as audio or subtitles, for example, by displaying them as subtitles to enable the user to converse smoothly (specific operation).

[0848] Step 11:

[0849] When the user inputs a prompt sentence, the terminal transmits the text data or voice data to the server (input).

[0850] Step 12:

[0851] The server uses a generative AI model to parse the prompt (input) and generate an appropriate response (output).

[0852] Step 13:

[0853] The server generates a response and sends it to the terminal (input), which may contain, for example, audio or text data (output).

[0854] Step 14:

[0855] The device then provides the generated response to the user via voice or text. For example, in response to the prompt, "Please tell me some recommended restaurants near Shinjuku Station," the device will provide information about recommended restaurants via voice (specific action).

[0856] (Application example 1)

[0857] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0858] In logistics centers, workers often face communication barriers when they need to efficiently receive work instructions over a wide area or when they need to use a foreign language. In addition, there is a need for real-time detection and response to work instructions and dangerous situations within large-scale facilities. These challenges must be addressed.

[0859] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0860] In this invention, the server includes a means for acquiring location information, a means for searching for and generating information about the surrounding area based on the acquired location information, a means for displaying the generated information, and a means for acquiring and displaying work instructions. This enables workers at a logistics center to efficiently receive instructions and perform appropriate work. Furthermore, a system including a means for capturing audio in different languages, a means for translating the captured audio into a specified language, a means for providing the translated results, and a means for displaying the translated results as support information facilitates communication between workers. Furthermore, a system including a means for capturing video of the surrounding area using a 360-degree camera, a means for analyzing the captured video to detect dangerous situations, a means for alerting users of detected danger information, and a means for improving work efficiency at logistics facilities enables improved safety and optimized work efficiency.

[0861] A "means for acquiring location information" is a device, equipment, or software that uses a GPS sensor or other location detection technology to identify and collect information about a person's current location.

[0862] "Means for searching and generating surrounding information" refers to a program or device that searches a database for information about the relevant area or location based on the acquired location information and generates the necessary information.

[0863] "Means for displaying generated information" refers to devices such as displays and AR / VR equipment that visually provide workers or users with the collected and generated information, or software for such devices.

[0864] "Means for obtaining and displaying work instructions" refers to devices or software that include a display or visual presentation method for obtaining specific work instructions from a server or other information source and displaying them in an easy-to-understand manner to the user.

[0865] The "means for capturing audio in different languages" is a microphone and audio collection software for collecting audio data in the languages ​​spoken by the user.

[0866] The "means for translating the captured voice into a specified language" is a translation engine or translation program for converting the captured voice data into another language.

[0867] "Means for providing translated results" means a display or audio player, or software for presenting the translated text or audio to the user.

[0868] The "means for displaying the translation results as support information" refers to a visual presentation device or interface for providing the translated text or audio to the user as work instructions or support information.

[0869] "Means for capturing surrounding images using a 360-degree camera" refers to a 360-degree camera for collecting surrounding images in all directions or an image collection device for such a purpose.

[0870] The "means for analyzing captured video to detect dangerous situations" refers to image analysis software or a program for analyzing acquired video data and detecting dangerous situations or abnormalities.

[0871] "Means for alerting the user to detected danger information" refers to a warning display device or audio alert device, or software for such devices, that notifies the user of detected danger or abnormality.

[0872] "Means for improving work efficiency at logistics facilities" refers to devices and software that provide users with appropriate work instructions and support information to make work within logistics facilities more efficient and smooth the flow of work.

[0873] MODE FOR CARRYING OUT THE INVENTION

[0874] A specific embodiment of the present invention will be described below. This embodiment is a system that uses smart glasses to improve work efficiency in a logistics center.

[0875] Overall system overview

[0876] The system consists of the following main components:

[0877] 1. Smart glasses (terminal)

[0878] 2. Server (backend system)

[0879] 3. Network Communication Infrastructure

[0880] Smart Glasses Configuration

[0881] The smart glasses have the following features:

[0882] GPS sensor: A sensor for obtaining location information.

[0883] 360-degree camera: A camera for capturing images of the surroundings.

[0884] Microphone: A device for capturing sound.

[0885] Speaker: A device for outputting sound.

[0886] Display: A display that can be displayed in AR / VR mode.

[0887] Network communication module: A module for communicating with the server.

[0888] Server Configuration

[0889] The server has the following features:

[0890] Database management system: Manages location-based surrounding information and work instructions.

[0891] Translation engine: Translates audio between different languages.

[0892] Video analysis engine: Analyzes video captured by a 360-degree camera to detect dangerous situations.

[0893] Information services: Providing necessary information such as weather forecasts and logistics information.

[0894] Information acquisition and provision process

[0895] 1. Obtaining location information

[0896] The GPS sensor in the smart glasses captures current location information and sends it to a server.

[0897] 2. Information retrieval and generation

[0898] Based on the received location information, the server searches the database for work instructions within the logistics center and generates optimized information.

[0899] 3. Displaying Information

[0900] Work instructions and surrounding information are displayed in AR on the smart glasses display and provided to workers, while information is also provided via audio.

[0901] 4. Voice translation function

[0902] When a worker initiates a conversation with a colleague who speaks a foreign language, the smart glasses' microphone captures the audio and sends it to the server, where the translation engine translates the speech and sends the results back to the smart glasses, where the translation results are displayed as subtitles on the display.

[0903] 5. Detecting dangerous situations

[0904] The 360-degree camera captures the surroundings in real time and sends it to a server. The server uses a video analytics engine to detect dangerous situations and sends that information to the smart glasses, which then provide a warning via AR display and audio.

[0905] Specific examples

[0906] For example, if a user working in a logistics center wants to know the location of a specific shelf, the smart glasses will obtain the location information and send it to the server. The server will then retrieve the location information from the database and display it on the user's glasses in AR. Furthermore, when the same user communicates with a worker who speaks a foreign language, the glasses can capture their voice and display the translation results on the display, enabling smooth conversation.

[0907] Example prompts to input to a generative AI model:

[0908] "You work at a logistics center in Tokyo. Please use the information below to navigate to the appropriate shelves and communicate with the foreign workers. Your current location is 35.6895, 139.6917."

[0909] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0910] Step 1:

[0911] The device's GPS sensor acquires the current location information. The input is location information, and the output is the acquired location data. The location data acquired from the GPS sensor is sent to the server.

[0912] Step 2:

[0913] The server analyzes the received location data and searches the database for work instructions within the logistics center based on that location information. The input is location data, and the output is the corresponding work instruction data. The database management system is used to generate the optimal work instructions.

[0914] Step 3:

[0915] The server sends the generated work instructions to the terminal. The terminal's display provides the work instructions to the user in an AR display. The input is the work instruction data, and the output is the AR display and audio instructions. The display and speaker are used to provide information visually and audibly.

[0916] Step 4:

[0917] The device's microphone captures audio in different languages. The input is audio data, and the output is the captured audio file. The audio data is sent to the server.

[0918] Step 5:

[0919] The server's translation engine analyzes the captured voice data and translates it into the specified language. The input is the voice data, and the output is the translated text data. The translation engine is used to generate an accurate translation.

[0920] Step 6:

[0921] The server sends the translation results to the terminal, and the terminal's display displays the translation results as subtitles. The input is the translated text data, and the output is the subtitle display. Visual information is provided to the user through the display.

[0922] Step 7:

[0923] The device's 360-degree camera captures the surroundings. The input is the video data, and the output is the captured video file. The video data is sent to the server.

[0924] Step 8:

[0925] The server's video analysis engine analyzes the captured video data and detects dangerous situations. The input is the video data and the output is the detected danger information. The video analysis engine is used to identify dangerous situations.

[0926] Step 9:

[0927] The server sends the detected danger information to the terminal, and the terminal's display and speaker provide a warning to the user. The input is the detected danger information, and the output is a warning display and audio warning. The warning is conveyed to the user using the display and speaker.

[0928] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0929] The smart ARVR glasses system of the present invention is a multifunctional device that allows users to obtain various information in real time, enabling efficient communication and work support. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusting the information provided according to the user's emotions, it can provide a more personalized user experience.

[0930] System configuration

[0931] The system consists of the following main components:

[0932] 1. Device (Smart ARVR Glasses)

[0933] GPS sensor

[0934] 360-degree camera

[0935] microphone

[0936] speaker

[0937] Display (AR / VR mode switchable)

[0938] Network Communication Module

[0939] Emotion Engine

[0940] 2. Server

[0941] Database Management Systems

[0942] Translation Engine

[0943] Video Analysis Engine

[0944] Sentiment Analysis Engine

[0945] Various information services such as weather forecasts, restaurant information, and tourist information

[0946] Information acquisition and provision process

[0947] When a user puts on the smart ARVR glasses and begins using them, the device first obtains its current location using the GPS sensor. This location information is then sent to the server via the network communication module. The server then uses the received location information to search its database for information on nearby restaurants, tourist spots, dangerous roads, weather forecasts, and other information, generating optimized information. The generated information is then sent to the device and provided to the user via voice or an AR screen.

[0948] Emotion Engine Functions

[0949] The emotion engine recognizes the user's current emotion from their voice, facial expressions, and actions, and uses the results to adjust the behavior of the entire system. For example, if the user is feeling stressed, the system will adjust to provide more concise and friendly information.

[0950] Machine translation function

[0951] When a user starts a conversation with someone speaking a foreign language, the device's microphone captures the voice, and the captured voice data is sent to the server along with emotional information. The server then passes the voice data and emotional information through a translation engine to translate the speech according to the user's emotions. The translation results are then sent to the device and provided to the user as audio or subtitles.

[0952] Check road conditions in real time and avoid accidents

[0953] While the user is moving, the device's 360-degree camera captures images of the surroundings in real time. This video data, along with emotional information, is sent to a server, which then uses a video analysis engine to detect dangerous situations and potential accidents. The detection results are then sent to the device, and a warning is displayed on the AR screen or via audio based on the user's emotional state.

[0954] Work support

[0955] When a user arrives at a designated work area at a manufacturing or logistics site, the device transmits location information and emotion information to a server. The server searches a database for relevant work procedures based on the location and emotion information, and transmits the retrieved work procedures to the device. The device then displays the procedures in AR, adjusted according to the user's emotion.

[0956] Customer service support

[0957] When a user starts a conversation with a customer while serving them, the device uses its voice recognition function to capture the conversation. The captured conversation data is sent to a server along with emotional information, and an analysis engine generates an optimal response example. The generated response example is sent to the device and provided to the user as audio or subtitles.

[0958] English conversation lessons

[0959] When a user starts an English conversation lesson, the device switches to VR mode and displays a generated AI model avatar. The user converses with the avatar, and their utterances and emotional information are captured by the device and sent to the server. The server analyzes the utterances and emotional information, generates appropriate responses, and sends them to the device. The device then provides the user with responses tailored to their emotions through the avatar.

[0960] Specific examples

[0961] When a user is walking near Shinjuku Station, the device acquires location information and sends it to the server. The server searches for recommended restaurant information and weather information around Shinjuku Station, generates optimal information based on the location information, and sends it to the device. The device then announces, "Here are some recommended restaurants," and displays the information on the AR screen. If the user shows signs of fatigue, the system also suggests nearby rest areas and coffee shops.

[0962] In addition, if a user is conversing in English with a foreign tourist, the device captures the conversation and sends it to the server. The server translates the English into Japanese and sends the translation result to the device. The device then displays the translated Japanese as subtitles, allowing the user to communicate smoothly. Furthermore, if the user is feeling tense, the system provides translation results using shorter sentences and simpler phrases.

[0963] As described above, the smart ARVR glasses system of the present invention integrates multiple functions, such as location information, voice capture, translation, video analysis, and emotion recognition, to provide users with a wide range of information support in real time. Furthermore, by incorporating an emotion engine, it is possible to flexibly respond to the user's emotional state, realizing a personalized user experience.

[0964] The processing flow will be explained below.

[0965] Information acquisition and provision process

[0966] Step 1:

[0967] Device: Uses the GPS sensor to obtain current location information.

[0968] Step 2:

[0969] Terminal: Sends the acquired location information to the server via the network communication module.

[0970] Step 3:

[0971] Server: Based on the received location information, the server searches a database for information such as nearby restaurants, tourist spots, dangerous roads, and weather forecasts.

[0972] Step 4:

[0973] Server: Optimizes search results and generates the information provided.

[0974] Step 5:

[0975] Server: Sends the generated information to the terminal.

[0976] Step 6:

[0977] Terminal: Provides received information to the user via voice or an AR screen.

[0978] Emotion Engine Functions

[0979] Step 1:

[0980] Device: Uses a microphone and camera to capture the user's voice and facial expressions.

[0981] Step 2:

[0982] Terminal: Sends captured audio and video data to the server.

[0983] Step 3:

[0984] Server: Recognizes user emotions using an emotion analysis engine.

[0985] Step 4:

[0986] Server: Adjust how information is presented based on perceived emotions.

[0987] Machine translation function

[0988] Step 1:

[0989] User: Start a conversation with someone who speaks a foreign language.

[0990] Step 2:

[0991] Device: Uses the microphone to capture audio.

[0992] Step 3:

[0993] Device: Sends captured audio data to the server.

[0994] Step 4:

[0995] Server: The voice data is passed through a translation engine and translated into the specified language.

[0996] Step 5:

[0997] Server: Adjusts translation results taking into account the user's emotional information.

[0998] Step 6:

[0999] Server: Sends the translation results to the device.

[1000] Step 7:

[1001] Device: Providing translated audio and subtitles to users.

[1002] Check road conditions in real time and avoid accidents

[1003] Step 1:

[1004] Device: Uses a 360-degree camera to capture footage of the surroundings in real time.

[1005] Step 2:

[1006] Terminal: Sends captured video data to the server.

[1007] Step 3:

[1008] Server: Uses a video analytics engine to detect dangerous situations and potential accidents.

[1009] Step 4:

[1010] Server: Adjust the warning content taking into account the user's emotional information.

[1011] Step 5:

[1012] Server: Sends the detection results to the device.

[1013] Step 6:

[1014] Device: Provides warning information to users via voice or AR screen.

[1015] Work support

[1016] Step 1:

[1017] User: Arrives at a designated work area on a manufacturing or logistics site.

[1018] Step 2:

[1019] Device: Obtains current location information and sends it to the server.

[1020] Step 3:

[1021] Server: Based on the location information, it searches the database for work procedures relevant to that area.

[1022] Step 4:

[1023] Server: Recognizes user emotions using an emotion analysis engine.

[1024] Step 5:

[1025] Server: Adjusts work procedures based on perceived emotions.

[1026] Step 6:

[1027] Server: Sends the adjusted work instructions to the device.

[1028] Step 7:

[1029] Terminal: Work procedures are displayed in AR in the user's field of view.

[1030] Customer service support

[1031] Step 1:

[1032] User: Starts a conversation with a customer.

[1033] Step 2:

[1034] Device: Use voice recognition to capture what you say.

[1035] Step 3:

[1036] Terminal: Sends captured conversation data to the server.

[1037] Step 4:

[1038] Server: Analyzes conversation data and generates optimal response examples.

[1039] Step 5:

[1040] Server: Recognizes user emotions using an emotion analysis engine.

[1041] Step 6:

[1042] Server: Adjusts response examples based on the recognized emotion.

[1043] Step 7:

[1044] Server: Sends the adjusted response examples to the device.

[1045] Step 8:

[1046] Device: Provide users with examples of what works with audio and subtitles.

[1047] English conversation lessons

[1048] Step 1:

[1049] User: Switches to VR mode to begin an English conversation lesson.

[1050] Step 2:

[1051] Terminal: Display the generated AI model avatar.

[1052] Step 3:

[1053] User: Converse with an AI avatar.

[1054] Step 4:

[1055] Device: Captures user speech and emotion information and sends it to the server.

[1056] Step 5:

[1057] Server: Analyzes the user's comments and emotional information and generates an appropriate response.

[1058] Step 6:

[1059] Server: Sends the adjusted response to the device.

[1060] Step 7:

[1061] Terminal: Provides tailored responses to users through avatars.

[1062] Example 2

[1063] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1064] Conventional information provision systems can only provide limited information based on the user's location information, making it difficult to respond flexibly to the user's emotions and situation. Furthermore, communication support in different languages ​​and detection and warning of dangerous situations can only respond uniformly without considering the user's emotions, so there is a need to improve the user experience.

[1065] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1066] In this invention, the server includes a means for acquiring location information, a means for searching and generating surrounding information based on the acquired location information, a means for displaying the generated information, and a means for recognizing the user's emotions and adjusting the information according to the emotions. This makes it possible to not only provide surrounding information based on the user's location information, but also to provide appropriate information, translation, and warnings according to the user's emotions.

[1067] "Location Information" means data of a user's current physical geographic coordinates obtained using a GPS sensor or other location detection device.

[1068] "Nearby information" refers to information related to the user's current location, such as nearby restaurants, tourist attractions, and weather information.

[1069] "Means for recognizing emotions" refers to technology that analyzes the user's voice, facial expressions, and movement data to estimate the user's emotional state.

[1070] A "360-degree camera" is a camera device capable of shooting in all directions, capturing a wide range of images at once.

[1071] A "dangerous situation" is a situation in the surroundings that may affect the user's safety, such as an accident or an obstacle.

[1072] "Adjusting according to emotions" means appropriately changing the information and warning content provided based on the user's current emotional state.

[1073] "Speech in a different language" refers to speech data in a language other than the language spoken by the user.

[1074] "Means for translating" refers to technology for converting audio data provided in a particular language into another language.

[1075] "Generated information" refers to data such as restaurant information and warnings generated based on location information, surrounding conditions, and the user's emotional state.

[1076] "Means of providing to the user" refers to the technology or device used to convey the generated information to the user in the form of audio, AR screen, text, etc.

[1077] The smart ARVR glasses system of the present invention is a multifunctional device that allows users to obtain various information in real time, enabling efficient communication and work support. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusting the information provided according to the user's emotions, it can provide a more personalized user experience.

[1078] System configuration

[1079] The system consists of the following main components:

[1080] 1. Device (Smart ARVR Glasses)

[1081] GPS sensor

[1082] 360-degree camera

[1083] microphone

[1084] speaker

[1085] Display (AR / VR mode switchable)

[1086] Network Communication Module

[1087] Emotion Engine

[1088] 2. Server

[1089] Database Management Systems

[1090] Translation Engine

[1091] Video Analysis Engine

[1092] Sentiment Analysis Engine

[1093] Various information services such as weather forecasts, restaurant information, and tourist information

[1094] Information acquisition and provision process

[1095] When a user puts on the smart ARVR glasses and begins using them, the device first obtains its current location using the GPS sensor. This location information is then sent to the server via the network communication module. The server then uses the received location information to search a database for information such as nearby restaurants, tourist spots, road hazards, and weather forecasts, and generates optimized information. The generated information is then sent to the device and provided to the user via voice or an AR screen.

[1096] Emotion Engine Functions

[1097] The emotion engine recognizes the user's current emotion from their voice, facial expressions, and actions, and uses the results to adjust the behavior of the entire system. For example, if the user is feeling stressed, the system will adjust to provide more concise and friendly information.

[1098] Machine translation function

[1099] When a user starts a conversation with someone speaking a foreign language, the device's microphone captures the voice, and the captured voice data is sent to the server along with emotional information. The server then passes the voice data and emotional information through a translation engine to translate the speech according to the user's emotions. The translation results are then sent to the device and provided to the user as audio or subtitles.

[1100] Check road conditions in real time and avoid accidents

[1101] While the user is moving, the device's 360-degree camera captures images of the surroundings in real time. This video data, along with emotional information, is sent to a server, which then uses a video analysis engine to detect dangerous situations and potential accidents. The detection results are then sent to the device, and a warning is displayed on the AR screen or via audio based on the user's emotional state.

[1102] Work support

[1103] When a user arrives at a designated work area at a manufacturing or logistics site, the device transmits location information and emotion information to a server. The server searches a database for relevant work procedures based on the location and emotion information, and transmits the retrieved work procedures to the device. The device then displays the procedures in AR, adjusted according to the user's emotion.

[1104] Customer service support

[1105] When a user starts a conversation with a customer while serving them, the device uses its voice recognition function to capture the conversation. The captured conversation data is sent to a server along with emotional information, and an analysis engine generates an optimal response example. The generated response example is sent to the device and provided to the user as audio or subtitles.

[1106] English conversation lessons

[1107] When a user starts an English conversation lesson, the device switches to VR mode and displays a generated AI model avatar. The user converses with the avatar, and their utterances and emotional information are captured by the device and sent to the server. The server analyzes the utterances and emotional information, generates appropriate responses, and sends them to the device. The device then provides the user with responses tailored to their emotions through the avatar.

[1108] Specific examples

[1109] For example, if a user is walking near Shinjuku Station, the device acquires location information and sends it to the server. The server searches for recommended restaurant information and weather information around Shinjuku Station, generates optimal information based on the location information, and sends it to the device. The device then announces, "Here are some recommended restaurants," and displays the information on the AR screen. If the user shows signs of fatigue, the system will also suggest nearby rest areas and coffee shops.

[1110] In addition, if a user is conversing in English with a foreign tourist, the device captures the conversation and sends it to the server. The server translates the English into Japanese and sends the translation result to the device. The device then displays the translated Japanese as subtitles, allowing the user to communicate smoothly. Furthermore, if the user is feeling tense, the system provides translation results using shorter sentences and simpler phrases.

[1111] Prompt Sentence Examples

[1112] An example prompt using a generative AI model would be:

[1113] "Please explain the process of providing information when a user walking near Shinjuku Station wants to check nearby restaurant recommendations. Also, please provide a specific example of how to respond if the user feels fatigued."

[1114] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1115] Step 1:

[1116] Acquiring and sending location information

[1117] The device uses the GPS sensor to obtain its current location.

[1118] Input: The user's current location.

[1119] Output: Real-time location data.

[1120] The acquired location information is transmitted to a server via a network communication module.

[1121] Operation: The device will announce "Current location obtained" and send data such as "Location information: Near Shinjuku Station" to the server.

[1122] Step 2:

[1123] Information Retrieval and Optimization

[1124] Based on the received location information, the server searches its database for nearby restaurants, tourist attractions, weather information, dangerous road information, and more.

[1125] Input: Location information sent from the device.

[1126] Output: Neighborhood information data as search results.

[1127] The server optimizes the acquired information and generates information to be presented to the user.

[1128] What it does: The server sends a query to the database to search for "recommended restaurants around Shinjuku Station." Based on the search results, it logs "Restaurant information generated."

[1129] Step 3:

[1130] Information provision and user emotional response

[1131] The device receives the generated information and provides it to the user via AR screen and audio.

[1132] Input: Optimized information data sent from the server.

[1133] Output: Visual and audio information displayed and notified to the user.

[1134] The device captures the user's reactions and emotions to the information provided.

[1135] How it works: The device will say "Here's a recommended restaurant" and display the restaurant's location on the AR screen. The device will capture the user's reaction and store it as emotion information.

[1136] Step 4:

[1137] Emotion Engine Operation

[1138] The server receives the user's voice, facial expression, and movement data and analyzes it using an emotion analysis engine.

[1139] Input: User voice, facial expression, and movement data sent from the device.

[1140] Output: Sentiment analysis result data.

[1141] The next information provided is adjusted according to the user's emotions.

[1142] Operation: If data is captured showing that the user is feeling tired, the emotion analysis engine detects "fatigue" and executes a process to "suggest nearby rest areas." Related information is sent to the device.

[1143] Step 5:

[1144] Machine translation function

[1145] The device uses a microphone to capture the user's voice and transmits it to the server along with emotional information.

[1146] Input: User's voice data and emotion information.

[1147] Output: Captured audio data and emotion information.

[1148] The server uses a translation engine to translate the captured audio.

[1149] Input: Voice data and emotion information sent from the device.

[1150] Output: Translation result data.

[1151] The system adjusts the translation based on the user's emotions and sends the translation results to the device.

[1152] Operation: The device notifies the user that "Audio data has been sent for translation," and the server logs "English to Japanese translation completed," and sends the translation results to the device.

[1153] Step 6:

[1154] Real-time road conditions and alerts

[1155] The device uses a 360-degree camera to capture images of the surroundings and send them to a server.

[1156] Input: Captured ambient video data and emotion information.

[1157] Output: Real-time video data.

[1158] The server uses a video analytics engine to detect dangerous situations.

[1159] Input: Video data sent from the device.

[1160] Output: Hazardous situation detection data.

[1161] The detection results are sent to the terminal and a warning is issued to the user.

[1162] What it does: The device displays "Capturing real-time video," the server logs "Dangerous intersection detected," and sends a warning to the device saying "Be careful."

[1163] Step 7:

[1164] Work support

[1165] When the user arrives at the work area, the device transmits location information and emotion information to the server.

[1166] Input: Location information obtained in the working area and user emotion information.

[1167] Output: Sending data to the server.

[1168] The server searches a database for relevant work procedures based on the location information and emotion information.

[1169] Input: Location and emotion information sent from the device.

[1170] Output: The searched work procedure data.

[1171] The acquired work procedures are sent to the terminal, and procedures adjusted according to the user's emotions are displayed in AR in the user's field of view.

[1172] Operation: The device issues a voice notification saying "Arrived at work area" and sends its location information to the server. The server logs "Work procedure generated" and sends it to the device.

[1173] Step 8:

[1174] Customer service support

[1175] The device uses voice recognition to capture conversations with customers.

[1176] Input: Customer conversation audio data.

[1177] Output: Captured conversation data and emotion information.

[1178] The server uses an analysis engine to generate the best possible response.

[1179] Input: Conversational data and emotional information sent from the device.

[1180] Output: Generated example data.

[1181] The generated examples are sent to the terminal and provided to the user as audio and subtitles.

[1182] How it works: The device displays "Capturing conversation," the server logs "Suggested response generated," and sends a suggestion to the device saying, "Here's what to say next."

[1183] Step 9:

[1184] English conversation lessons

[1185] The device switches to VR mode and displays the generated AI model avatar. The user's speech and emotional information are sent to the server.

[1186] Input: User utterance data and emotion information.

[1187] Output: Captured speech data and emotion information.

[1188] The server analyzes the content of the statement and emotional information and generates an appropriate response.

[1189] Input: Speech data and emotion information sent from the device.

[1190] output: The generated response data.

[1191] The generated response is sent to the terminal and provided to the user through the avatar.

[1192] Operation: The device announces "Start English conversation lesson" and displays an avatar. The server logs "Analyzing user's comment" and generates an appropriate response and sends it to the device.

[1193] The above are the specific processing steps of the program of this system.

[1194] (Application example 2)

[1195] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1196] Conventional ARVR systems and information acquisition systems provide information uniformly without considering the user's state or emotions, making it difficult to achieve a personalized user experience. While efficient real-time information provision is particularly required in logistics centers and factories, conventional systems have difficulty responding flexibly to the worker's emotions and situation. Furthermore, smooth communication between different languages ​​is difficult because translation results are provided without taking emotions into account. Furthermore, detection of dangerous situations and warnings during work do not reflect emotional information, making it difficult to respond appropriately to the user's state.

[1197] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring location information, means for displaying generated information, means for recognizing emotions, means for adjusting information to be provided according to the recognized emotions, means for capturing voices in different languages, means for providing translation results, means for translating the captured voices into a specified language, means for capturing surrounding video using a 360-degree camera, means for analyzing the captured video to detect dangerous situations, means for alerting the user of detected danger information, and means for acquiring emotional information and adjusting the content of the alert based on the analysis results. This makes it possible to provide personalized information based on the user's location information and emotional state and improve safety.

[1198] 1. "Means for obtaining location information" refers to a device that uses a GPS sensor or other location-determining technology to determine the user's current location.

[1199] 2. "Means for searching and generating surrounding information" refers to a processing system that searches a database for information about the surrounding environment and available services based on location information and provides it to the user.

[1200] 3. "Means for displaying generated information" refers to devices such as displays and speakers that provide acquired information to the user visually or audibly.

[1201] 4. "Means for recognizing emotions" refers to software and sensors for detecting and analyzing the user's emotional state from their voice, facial expressions, movements, etc.

[1202] 5. "Means for adjusting the information provided in response to the recognized emotion" means a processing system that changes the content or format of the information provided based on the detected emotion information.

[1203] 6. "Means for capturing audio in different languages" means microphones and speech analysis devices for collecting audio data corresponding to multiple languages.

[1204] 7. "Means for translating captured audio into a specified language" means translation software or services that convert captured audio data into another language.

[1205] 8. "Means for providing translation results" means a display or speaker for providing the translated audio or text to the user.

[1206] 9. "Means for capturing surrounding images using a 360-degree camera" means a camera device for capturing images in all directions.

[1207] 10. "Means for analyzing captured video to detect dangerous situations" means software that analyzes captured video data and recognizes dangerous situations and obstacles.

[1208] 11. "Means for warning the user of detected danger information" refers to a device that issues a visual or audible warning to the user based on the analysis results.

[1209] 12. "Means for acquiring emotional information and adjusting the content of warnings based on the analysis results" refers to a processing system for changing the content and format of warnings based on the user's emotional state.

[1210] Overall system overview

[1211] The system of this invention uses smart ARVR glasses to provide information and support to logistics center workers. This system acquires location information, analyzes emotional information, and provides the generated information to users, enabling efficient and safe work.

[1212] Hardware Configuration

[1213] 1. Device (Smart ARVR Glasses):

[1214] GPS sensor: Obtain location information

[1215] 360-degree camera: Capture footage of your surroundings

[1216] Microphone, speaker: Audio capture and output

[1217] Display (AR / VR mode switchable)

[1218] Network communication module: Data communication with the server

[1219] Emotion engine: Recognizes user emotions and adjusts information delivery accordingly

[1220] 2. Server:

[1221] Database Management System (DBMS): MariaDB

[1222] Translation engine: Google Cloud Translation API

[1223] Video analysis engine: OpenCV

[1224] Sentiment analysis engine: IBM Watson Tone Analyzer

[1225] Various information service APIs (work procedures, hazard information, etc.)

[1226] Software Configuration and Processing

[1227] The server processes information using the following means.

[1228] 1. Obtaining and sending location information:

[1229] The device's GPS sensor acquires location information and transmits it to the server via the network communication module. The server then references the database based on the location information to search for and generate relevant work procedures.

[1230] 2. Acquisition and analysis of emotional information:

[1231] The emotion engine captures the user's voice and facial expressions and sends them to the emotion analysis engine. The server analyzes the emotional information and generates optimal information according to the user's state.

[1232] 3. Video capture and analysis:

[1233] The 360-degree camera captures the surroundings and sends the video data to a server, which uses a video analytics engine to detect dangerous situations and alert the user.

[1234] Specific examples

[1235] 1. Display of work procedure:

[1236] When the user arrives at the designated work area, the device acquires the user's location information and sends it to the server. The server searches the database for relevant work procedures based on the location information, and sends the acquired work procedures to the device. The device then displays the procedures in AR, adjusted according to the user's emotions.

[1237] Example prompt: Tell me the procedure, and the location is "34.052235, -118.243683"

[1238] 2. Notification of dangerous situations:

[1239] While the user is moving, a 360-degree camera captures the surroundings and sends it to a server. The server uses a video analysis engine to detect dangerous situations and warn the user. Emotional information is also analyzed, and the warning content is adjusted accordingly.

[1240] Example prompt: Tells about danger around you and makes you feel nervous

[1241] 3. Multilingual Translation:

[1242] When a user starts a conversation with someone who speaks a different language, the device's microphone captures the voice and sends it along with emotional information to the server. The server then uses a translation engine to translate the voice into the specified language and sends the translation result to the device. The translation result is adjusted appropriately according to the user's emotional state.

[1243] Example prompt: Translate the conversation and the emotion is "stress"

[1244] As a result, the present invention enables workers in a logistics center to receive personalized support and perform their work safely and efficiently.

[1245] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1246] Program processing steps

[1247] Process flow for displaying work procedures

[1248] Step 1:

[1249] The device's GPS sensor acquires the user's current location. The input is location data from the GPS sensor, and the output is the acquired location information. Specifically, the GPS sensor periodically acquires location data and stores it in internal memory.

[1250] Step 2:

[1251] The location information acquired by the terminal is sent to the server via the network communication module. The input is location information, and the output is location data sent to the server. Specifically, the terminal converts the location information into packets and sends them to the server via the network communication module over the Internet.

[1252] Step 3:

[1253] The server refers to the database management system (DBMS) based on the location information it receives and searches for related work procedures. The input is the location information received by the server, and the output is the searched work procedure data. Specifically, the server generates an SQL query and retrieves related work procedures from the database.

[1254] Step 4:

[1255] The server sends the retrieved work procedure to the terminal via the network communication module. The input is the retrieved work procedure data, and the output is the work procedure to be sent to the terminal. Specifically, the server converts the work procedure data into packets and sends them to the terminal via the network communication module.

[1256] Step 5:

[1257] The device's emotion engine analyzes the user's voice and facial expressions to obtain emotional information. The input is voice data and facial expression data, and the output is emotional information. Specifically, the device captures voice and facial expressions using a microphone and camera, and processes them using the built-in emotion analysis engine.

[1258] Step 6:

[1259] The device adjusts the displayed work steps based on the emotional information it acquires. The input is emotional information and work step data, and the output is the adjusted work steps. Specifically, the emotion engine analyzes the emotional information and changes the display format and content to suit the user's state.

[1260] Step 7:

[1261] The device displays the adjusted work procedure in AR in the user's field of view. The input is the adjusted work procedure data, and the output is visual information provided to the user. Specifically, the device display displays the adjusted work procedure in AR mode, allowing the user to visually confirm it.

[1262] Processing flow for notification of dangerous situations

[1263] Step 1:

[1264] The device's 360-degree camera captures the surroundings. The input is the camera's video data, and the output is the captured video data. Specifically, the camera periodically captures video and stores it in its internal memory.

[1265] Step 2:

[1266] The video data captured by the terminal is sent to the server via the network communication module. The input is video data, and the output is video data sent to the server. Specifically, the terminal converts the video data into packets and sends them to the server via the network.

[1267] Step 3:

[1268] The video data received by the server is analyzed by a video analysis engine to detect dangerous situations. The input is video data and the output is detected danger information. Specifically, the video analysis engine analyzes the video data and automatically detects dangerous situations.

[1269] Step 4:

[1270] The server sends danger information to the terminal via the network communication module. The input is danger information, and the output is danger data sent to the terminal. Specifically, the server converts the danger information into packets and sends them to the terminal via the network communication module.

[1271] Step 5:

[1272] The device's emotion engine analyzes the user's voice and facial expressions to obtain emotional information. The input is voice data and facial expression data, and the output is emotional information. Specifically, the device captures voice and facial expressions using a microphone and camera, and processes them using the built-in emotion analysis engine.

[1273] Step 6:

[1274] The device adjusts the warning content based on danger information and emotional information. The input is danger information and emotional information, and the output is the adjusted warning content. Specifically, the emotion engine analyzes the emotional information and changes the content and format of the warning to suit the user's state.

[1275] Step 7:

[1276] The device notifies the user of the adjusted warning visually or audibly. The input is the adjusted warning content, and the output is the warning information provided to the user. Specifically, the device's display or speaker displays or notifies the user of the adjusted warning, allowing the user to respond immediately.

[1277] The above are the specific processing steps for carrying out the invention.

[1278] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1279] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1280] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1281] [Third embodiment]

[1282] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1283] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1284] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1285] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1286] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1287] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1288] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1289] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1290] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1291] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1292] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1293] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1294] The smart ARVR glasses system of the present invention is a multi-functional device that allows users to obtain various information in real time and provide efficient communication and work assistance.

[1295] System configuration

[1296] The system consists of the following main components:

[1297] 1. Device (Smart ARVR Glasses)

[1298] GPS sensor

[1299] 360-degree camera

[1300] microphone

[1301] speaker

[1302] Display (AR / VR mode switchable)

[1303] Network Communication Module

[1304] 2. Server

[1305] Database Management Systems

[1306] Translation Engine

[1307] Video Analysis Engine

[1308] Various information services such as weather forecasts, restaurant information, and tourist information

[1309] Information acquisition and provision process

[1310] When a user puts on the smart ARVR glasses and begins using them, the device first obtains its current location using the GPS sensor. This location information is then sent to the server via the network communication module. The server then uses the received location information to search its database for information on nearby restaurants, tourist spots, dangerous roads, weather forecasts, and other information, generating optimized information. The generated information is then sent to the device and provided to the user via voice or an AR screen.

[1311] Machine translation function

[1312] When a user starts a conversation with someone who speaks a foreign language, the device's microphone captures the voice and sends the captured voice data to the server. The server then passes the voice data through a translation engine to translate it into the required language. The translation result is then sent to the device and provided to the user as audio or subtitles.

[1313] Check road conditions in real time and avoid accidents

[1314] While the user is moving, the device's 360-degree camera captures video of the surroundings in real time. This video data is sent to a server, which uses a video analytics engine to detect dangerous situations or potential accidents. The results are then sent to the device, and a warning is displayed to the user via audio and an AR screen.

[1315] Work support

[1316] When a user arrives at a designated work area at a manufacturing or logistics site, the device sends location information to a server. The server searches a database for work procedures related to that area based on the location information, and sends the retrieved work procedures to the device. The device then displays the procedures in AR within the user's field of view, helping to improve work efficiency.

[1317] Customer service support

[1318] When a user starts a conversation with a customer while serving them, the device uses its voice recognition function to capture the conversation. The captured conversation data is sent to a server, where an analysis engine generates the most appropriate response. The generated response is sent to the device and provided to the user as audio or subtitles.

[1319] English conversation lessons

[1320] When a user starts an English conversation lesson, the device switches to VR mode and displays a generated AI model avatar. The user converses with the avatar, and their utterances are captured by the device and sent to the server. The server analyzes the utterances, generates appropriate responses, and sends them to the device. The device then provides the generated responses to the user through the avatar.

[1321] Specific examples

[1322] When a user is walking near Shinjuku Station, the device acquires location information and sends it to the server. The server searches for recommended restaurants and weather information around Shinjuku Station, generates the most appropriate information, and sends it to the device. The device then announces, "Here are some recommended restaurants," and displays it on the AR screen.

[1323] In addition, if a user is conversing in English with a foreign tourist, the device captures the conversation and sends it to the server. The server translates the English into Japanese and sends the translation result to the device. The device displays the translated Japanese as subtitles, allowing the user to communicate smoothly.

[1324] As described above, the smart ARVR glasses system of the present invention can provide users with a wide range of information support in real time by integrating multiple functions such as location information, voice capture, translation, and video analysis.

[1325] The processing flow will be explained below.

[1326] Information acquisition and provision process

[1327] Step 1:

[1328] Device: Uses the GPS sensor to obtain current location information.

[1329] Step 2:

[1330] Terminal: Sends the acquired location information to the server via the network communication module.

[1331] Step 3:

[1332] Server: Based on the received location information, the server searches a database for information such as nearby restaurants, tourist spots, dangerous roads, and weather forecasts.

[1333] Step 4:

[1334] Server: Optimizes search results and generates the information provided.

[1335] Step 5:

[1336] Server: Sends the generated information to the terminal.

[1337] Step 6:

[1338] Terminal: Provides received information to the user via voice or an AR screen.

[1339] Machine translation function

[1340] Step 1:

[1341] User: Start a conversation with someone who speaks a foreign language.

[1342] Step 2:

[1343] Device: Uses the microphone to capture audio.

[1344] Step 3:

[1345] Device: Sends captured audio data to the server.

[1346] Step 4:

[1347] Server: The received voice data is passed through a translation engine and translated into the specified language.

[1348] Step 5:

[1349] Server: Sends the translation results to the device.

[1350] Step 6:

[1351] Device: Providing translated audio and subtitles to users.

[1352] Check road conditions in real time and avoid accidents

[1353] Step 1:

[1354] Device: Uses a 360-degree camera to capture footage of the surroundings in real time.

[1355] Step 2:

[1356] Terminal: Sends captured video data to the server.

[1357] Step 3:

[1358] Server: Analyzes the received video data and detects dangerous situations and potential accidents.

[1359] Step 4:

[1360] Server: Sends the detection results to the device.

[1361] Step 5:

[1362] Device: Provides warning information to users via voice or AR screen.

[1363] Work support

[1364] Step 1:

[1365] User: Arrives at a designated work area on a manufacturing or logistics site.

[1366] Step 2:

[1367] Device: Obtains current location information and sends it to the server.

[1368] Step 3:

[1369] Server: Based on the location information, it searches the database for work procedures relevant to that area.

[1370] Step 4:

[1371] Server: Sends search results to the device.

[1372] Step 5:

[1373] Terminal: Work procedures are displayed in AR in the user's field of view.

[1374] Customer service support

[1375] Step 1:

[1376] User: Starts a conversation with a customer.

[1377] Step 2:

[1378] Device: Use voice recognition to capture what you say.

[1379] Step 3:

[1380] Terminal: Sends captured conversation data to the server.

[1381] Step 4:

[1382] Server: Analyzes conversation data and generates optimal response examples.

[1383] Step 5:

[1384] Server: Sends the generated response examples to the device.

[1385] Step 6:

[1386] Device: Provide users with examples of what works with audio and subtitles.

[1387] English conversation lessons

[1388] Step 1:

[1389] User: Switches to VR mode to begin an English conversation lesson.

[1390] Step 2:

[1391] Terminal: Display the generated AI model avatar.

[1392] Step 3:

[1393] User: Converse with an AI avatar.

[1394] Step 4:

[1395] Terminal: Captures what the user says and sends it to the server.

[1396] Step 5:

[1397] Server: Analyzes what the user says and generates an appropriate response.

[1398] Step 6:

[1399] Server: Generates a response and sends it to the device.

[1400] Step 7:

[1401] Terminal: The generated response is presented to the user through an avatar.

[1402] Example 1

[1403] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1404] In conventional smart devices and information systems, functions such as location-based information provision, language translation, and hazard detection were implemented separately, forcing users to use multiple devices and applications simultaneously. Furthermore, because the various functions were not linked in real time, it was difficult to efficiently obtain information or provide work support. Therefore, a system that integrated all functions was needed.

[1405] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1406] In this invention, the server includes means for acquiring location information, means for searching for surrounding information based on the acquired location information, means for generating optimal information based on the searched information, means for displaying the generated information, means for capturing audio in different languages, means for translating the captured audio into a specified language, means for providing the translated result, means for capturing video, means for analyzing the captured video to detect a dangerous situation, means for alerting the user of detected danger information, means for inputting a prompt sentence, means for analyzing the input prompt sentence, means for generating an appropriate response based on the analysis result, and means for providing the generated response. This enables a single system to provide multiple functions including information provision, translation, danger detection, and conversation support based on prompts.

[1407] "Means for acquiring location information" refers to a device or system for measuring the current location and acquiring that information as digital data.

[1408] The "means for searching surrounding information" is a system for extracting information about the region and environment from a database based on the acquired location information.

[1409] "Means for generating optimal information" refers to algorithms or programs that evaluate the searched information, convert it into the most useful form for the user, and provide it to the user.

[1410] The "means for displaying information" refers to a device or interface for visually or audibly presenting the generated information to the user.

[1411] A "means for capturing audio in different languages" is a system for using a device such as a microphone to pick up audio spoken in different languages ​​and record it as digital data.

[1412] A "means for translating speech into a specified language" is software or algorithms that analyze the captured speech data and translate it into a target language.

[1413] "Means for providing translated results" refers to a function or device for outputting translated language data to the user in the form of voice or text.

[1414] A "means for capturing video" is a system that uses a device such as a camera to capture video of the surroundings and record it as digital data.

[1415] "Means for analyzing captured video to detect dangerous situations" refers to algorithms or programs that analyze the acquired video data and recognize dangerous situations or abnormal movements from it.

[1416] The "means for warning the user about danger information" refers to means such as sound, visual display, vibration, etc. for notifying the user of the detected danger information.

[1417] A "means for inputting prompt sentences" is an interface that allows the user to freely enter questions or instructions into the system through text input or voice input.

[1418] The "means for analyzing the input prompt sentence" is a natural language processing algorithm for understanding the prompt sentence input by the user and performing appropriate processing according to its content.

[1419] The "means for generating an appropriate response" is software or a model for generating an accurate and appropriate response to the user based on the analyzed prompt sentence.

[1420] The "means for providing a response" refers to a function or device that outputs the generated response to the user in voice or text form and supports the dialogue with the user.

[1421] The system of the present invention is an integrated information provision system that combines a smart device and a server. This makes it possible to provide multiple functions such as location information, translation, road condition confirmation, and prompt sentence analysis in an integrated manner. The specific configuration and operation procedure are described below.

[1422] System configuration

[1423] The system mainly consists of the following components:

[1424] 1. Terminal (smart device such as smart glasses)

[1425] GPS sensor

[1426] Cameras (including 360-degree cameras)

[1427] microphone

[1428] speaker

[1429] Display (AR / VR mode switchable)

[1430] Network communication module (Wi-Fi and mobile data communication)

[1431] 2. Server

[1432] Database Management Systems

[1433] Translation engine (e.g. Google Translate API)

[1434] Video Analysis Engine

[1435] information search engine

[1436] Generative AI Models

[1437] Main functions and processing flow

[1438] Information provision function

[1439] When a user puts on a smart device and starts using it, the device's GPS sensor acquires its current location. The acquired location information is sent to a server via a network communication module. Based on this location information, the server uses a database management system to search for information about the surrounding area and generate the most appropriate information from the search results. This information is then provided to the user in the form of audio, AR display, or other formats.

[1440] Specific examples

[1441] When a user is walking near Shinjuku Station, the device's GPS sensor acquires location information and sends it to the server. The server searches for recommended restaurants and weather information around Shinjuku Station, generates the most appropriate information, and sends it to the device. The device then announces, "Here are some recommended restaurants," and displays it on the AR screen.

[1442] Machine translation function

[1443] When a user starts a conversation with someone speaking a foreign language, the device's microphone captures the audio and sends it to the server. The server then translates the audio data into the required language through a translation engine and sends the translation results to the device. The translation results are then provided to the user as audio or subtitles.

[1444] Specific examples

[1445] When a user is conversing in English with a foreign tourist, the device's microphone captures the conversation and sends it to the server. The server then passes the voice data through a translation engine to translate the English into Japanese and sends the result to the device. The device then displays the translated Japanese as subtitles, allowing the user to communicate smoothly.

[1446] Road condition check function

[1447] While the user is moving, the device's 360-degree camera captures video of the surroundings. This video data is sent to a server, which uses a video analysis engine to detect dangerous situations or potential accidents. The results are then sent to the device, where they are used to warn the user via audio and AR displays.

[1448] Prompt sentence analysis function

[1449] When a user enters a prompt, the device sends the text or voice data to the server, which uses a generative AI model to analyze the prompt and generate an appropriate response, which is then sent to the device and provided to the user as voice or text.

[1450] Prompt Sentence Examples

[1451] "Please tell me some recommended restaurants near Shinjuku Station."

[1452] As described above, the system of the present invention provides multiple functions, including location information, voice capture, translation, video analysis, and prompt sentence analysis, in an integrated manner, and can provide multifunctional information support to users in real time.

[1453] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1454] Step 1:

[1455] The user wears a smart device and starts the system. The device acquires the current location information using the GPS sensor. This location information (input) is stored on the device as coordinate data (output).

[1456] Step 2:

[1457] The device sends the acquired location information to the server via the network communication module (input). At this time, the data is encrypted before being sent (specific operation).

[1458] Step 3:

[1459] The server uses the database management system to search for surrounding information based on the received location information (input), and extracts information about restaurants, tourist spots, dangerous road conditions, and other information from the database near the current location (output).

[1460] Step 4:

[1461] The server generates the most appropriate information for the user based on the search results (input). For example, if the user is a pedestrian, it will prioritize information about restaurants and weather within walking distance (output).

[1462] Step 5:

[1463] The server sends the generated optimized information to the terminal (input). The sent information includes audio data, text data, and video data (output).

[1464] Step 6:

[1465] The device then provides the received information to the user via voice or an AR screen. For example, it may say, "Here are some recommended restaurants," and display a map on the AR screen (specifically, an example of the area around Shinjuku Station).

[1466] Step 7:

[1467] When a user starts a conversation with someone who speaks a foreign language, the device's microphone captures the conversation (input) and sends the audio data to a server (output).

[1468] Step 8:

[1469] The server translates the received voice data using a translation engine (input). For example, to translate from English to Japanese, it uses the Google Translate API (output).

[1470] Step 9:

[1471] The server generates the translation result and sends it to the terminal (input), which contains the translated text data (output).

[1472] Step 10:

[1473] The device provides the translation results to the user as audio or subtitles, for example, by displaying them as subtitles to enable the user to converse smoothly (specific operation).

[1474] Step 11:

[1475] When the user inputs a prompt sentence, the terminal transmits the text data or voice data to the server (input).

[1476] Step 12:

[1477] The server uses a generative AI model to parse the prompt (input) and generate an appropriate response (output).

[1478] Step 13:

[1479] The server generates a response and sends it to the terminal (input), which may contain, for example, audio or text data (output).

[1480] Step 14:

[1481] The device then provides the generated response to the user via voice or text. For example, in response to the prompt, "Please tell me some recommended restaurants near Shinjuku Station," the device will provide information about recommended restaurants via voice (specific action).

[1482] (Application example 1)

[1483] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1484] In logistics centers, workers often face communication barriers when they need to efficiently receive work instructions over a wide area or when they need to use a foreign language. In addition, there is a need for real-time detection and response to work instructions and dangerous situations within large-scale facilities. These challenges must be addressed.

[1485] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1486] In this invention, the server includes a means for acquiring location information, a means for searching for and generating information about the surrounding area based on the acquired location information, a means for displaying the generated information, and a means for acquiring and displaying work instructions. This enables workers at a logistics center to efficiently receive instructions and perform appropriate work. Furthermore, a system including a means for capturing audio in different languages, a means for translating the captured audio into a specified language, a means for providing the translated results, and a means for displaying the translated results as support information facilitates communication between workers. Furthermore, a system including a means for capturing video of the surrounding area using a 360-degree camera, a means for analyzing the captured video to detect dangerous situations, a means for alerting users of detected danger information, and a means for improving work efficiency at logistics facilities enables improved safety and optimized work efficiency.

[1487] A "means for acquiring location information" is a device, equipment, or software that uses a GPS sensor or other location detection technology to identify and collect information about a person's current location.

[1488] "Means for searching and generating surrounding information" refers to a program or device that searches a database for information about the relevant area or location based on the acquired location information and generates the necessary information.

[1489] "Means for displaying generated information" refers to devices such as displays and AR / VR equipment that visually provide workers or users with the collected and generated information, or software for such devices.

[1490] "Means for obtaining and displaying work instructions" refers to devices or software that include a display or visual presentation method for obtaining specific work instructions from a server or other information source and displaying them in an easy-to-understand manner to the user.

[1491] The "means for capturing audio in different languages" is a microphone and audio collection software for collecting audio data in the languages ​​spoken by the user.

[1492] The "means for translating the captured voice into a specified language" is a translation engine or translation program for converting the captured voice data into another language.

[1493] "Means for providing translated results" means a display or audio player, or software for presenting the translated text or audio to the user.

[1494] The "means for displaying the translation results as support information" refers to a visual presentation device or interface for providing the translated text or audio to the user as work instructions or support information.

[1495] "Means for capturing surrounding images using a 360-degree camera" refers to a 360-degree camera for collecting surrounding images in all directions or an image collection device for such a purpose.

[1496] The "means for analyzing captured video to detect dangerous situations" refers to image analysis software or a program for analyzing acquired video data and detecting dangerous situations or abnormalities.

[1497] "Means for alerting the user to detected danger information" refers to a warning display device or audio alert device, or software for such devices, that notifies the user of detected danger or abnormality.

[1498] "Means for improving work efficiency at logistics facilities" refers to devices and software that provide users with appropriate work instructions and support information to make work within logistics facilities more efficient and smooth the flow of work.

[1499] MODE FOR CARRYING OUT THE INVENTION

[1500] A specific embodiment of the present invention will be described below. This embodiment is a system that uses smart glasses to improve work efficiency in a logistics center.

[1501] Overall system overview

[1502] The system consists of the following main components:

[1503] 1. Smart glasses (terminal)

[1504] 2. Server (backend system)

[1505] 3. Network Communication Infrastructure

[1506] Smart Glasses Configuration

[1507] The smart glasses have the following features:

[1508] GPS sensor: A sensor for obtaining location information.

[1509] 360-degree camera: A camera for capturing images of the surroundings.

[1510] Microphone: A device for capturing sound.

[1511] Speaker: A device for outputting sound.

[1512] Display: A display that can be displayed in AR / VR mode.

[1513] Network communication module: A module for communicating with the server.

[1514] Server Configuration

[1515] The server has the following features:

[1516] Database management system: Manages location-based surrounding information and work instructions.

[1517] Translation engine: Translates audio between different languages.

[1518] Video analysis engine: Analyzes video captured by a 360-degree camera to detect dangerous situations.

[1519] Information services: Providing necessary information such as weather forecasts and logistics information.

[1520] Information acquisition and provision process

[1521] 1. Obtaining location information

[1522] The GPS sensor in the smart glasses captures current location information and sends it to a server.

[1523] 2. Information retrieval and generation

[1524] Based on the received location information, the server searches the database for work instructions within the logistics center and generates optimized information.

[1525] 3. Displaying Information

[1526] Work instructions and surrounding information are displayed in AR on the smart glasses display and provided to workers, while information is also provided via audio.

[1527] 4. Voice translation function

[1528] When a worker initiates a conversation with a colleague who speaks a foreign language, the smart glasses' microphone captures the audio and sends it to the server, where the translation engine translates the speech and sends the results back to the smart glasses, where the translation results are displayed as subtitles on the display.

[1529] 5. Detecting dangerous situations

[1530] The 360-degree camera captures the surroundings in real time and sends it to a server. The server uses a video analytics engine to detect dangerous situations and sends that information to the smart glasses, which then provide a warning via AR display and audio.

[1531] Specific examples

[1532] For example, if a user working in a logistics center wants to know the location of a specific shelf, the smart glasses will obtain the location information and send it to the server. The server will then retrieve the location information from the database and display it on the user's glasses in AR. Furthermore, when the same user communicates with a worker who speaks a foreign language, the glasses can capture their voice and display the translation results on the display, enabling smooth conversation.

[1533] Example prompts to input to a generative AI model:

[1534] "You work at a logistics center in Tokyo. Please use the information below to navigate to the appropriate shelves and communicate with the foreign workers. Your current location is 35.6895, 139.6917."

[1535] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1536] Step 1:

[1537] The device's GPS sensor acquires the current location information. The input is location information, and the output is the acquired location data. The location data acquired from the GPS sensor is sent to the server.

[1538] Step 2:

[1539] The server analyzes the received location data and searches the database for work instructions within the logistics center based on that location information. The input is location data, and the output is the corresponding work instruction data. The database management system is used to generate the optimal work instructions.

[1540] Step 3:

[1541] The server sends the generated work instructions to the terminal. The terminal's display provides the work instructions to the user in an AR display. The input is the work instruction data, and the output is the AR display and audio instructions. The display and speaker are used to provide information visually and audibly.

[1542] Step 4:

[1543] The device's microphone captures audio in different languages. The input is audio data, and the output is the captured audio file. The audio data is sent to the server.

[1544] Step 5:

[1545] The server's translation engine analyzes the captured voice data and translates it into the specified language. The input is the voice data, and the output is the translated text data. The translation engine is used to generate an accurate translation.

[1546] Step 6:

[1547] The server sends the translation results to the terminal, and the terminal's display displays the translation results as subtitles. The input is the translated text data, and the output is the subtitle display. Visual information is provided to the user through the display.

[1548] Step 7:

[1549] The device's 360-degree camera captures the surroundings. The input is the video data, and the output is the captured video file. The video data is sent to the server.

[1550] Step 8:

[1551] The server's video analysis engine analyzes the captured video data and detects dangerous situations. The input is the video data and the output is the detected danger information. The video analysis engine is used to identify dangerous situations.

[1552] Step 9:

[1553] The server sends the detected danger information to the terminal, and the terminal's display and speaker provide a warning to the user. The input is the detected danger information, and the output is a warning display and audio warning. The warning is conveyed to the user using the display and speaker.

[1554] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1555] The smart ARVR glasses system of the present invention is a multifunctional device that allows users to obtain various information in real time, enabling efficient communication and work support. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusting the information provided according to the user's emotions, it can provide a more personalized user experience.

[1556] System configuration

[1557] The system consists of the following main components:

[1558] 1. Device (Smart ARVR Glasses)

[1559] GPS sensor

[1560] 360-degree camera

[1561] microphone

[1562] speaker

[1563] Display (AR / VR mode switchable)

[1564] Network Communication Module

[1565] Emotion Engine

[1566] 2. Server

[1567] Database Management Systems

[1568] Translation Engine

[1569] Video Analysis Engine

[1570] Sentiment Analysis Engine

[1571] Various information services such as weather forecasts, restaurant information, and tourist information

[1572] Information acquisition and provision process

[1573] When a user puts on the smart ARVR glasses and begins using them, the device first obtains its current location using the GPS sensor. This location information is then sent to the server via the network communication module. The server then uses the received location information to search its database for information on nearby restaurants, tourist spots, dangerous roads, weather forecasts, and other information, generating optimized information. The generated information is then sent to the device and provided to the user via voice or an AR screen.

[1574] Emotion Engine Functions

[1575] The emotion engine recognizes the user's current emotion from their voice, facial expressions, and actions, and uses the results to adjust the behavior of the entire system. For example, if the user is feeling stressed, the system will adjust to provide more concise and friendly information.

[1576] Machine translation function

[1577] When a user starts a conversation with someone speaking a foreign language, the device's microphone captures the voice, and the captured voice data is sent to the server along with emotional information. The server then passes the voice data and emotional information through a translation engine to translate the speech according to the user's emotions. The translation results are then sent to the device and provided to the user as audio or subtitles.

[1578] Check road conditions in real time and avoid accidents

[1579] While the user is moving, the device's 360-degree camera captures images of the surroundings in real time. This video data, along with emotional information, is sent to a server, which then uses a video analysis engine to detect dangerous situations and potential accidents. The detection results are then sent to the device, and a warning is displayed on the AR screen or via audio based on the user's emotional state.

[1580] Work support

[1581] When a user arrives at a designated work area at a manufacturing or logistics site, the device transmits location information and emotion information to a server. The server searches a database for relevant work procedures based on the location and emotion information, and transmits the retrieved work procedures to the device. The device then displays the procedures in AR, adjusted according to the user's emotion.

[1582] Customer service support

[1583] When a user starts a conversation with a customer while serving them, the device uses its voice recognition function to capture the conversation. The captured conversation data is sent to a server along with emotional information, and an analysis engine generates an optimal response example. The generated response example is sent to the device and provided to the user as audio or subtitles.

[1584] English conversation lessons

[1585] When a user starts an English conversation lesson, the device switches to VR mode and displays a generated AI model avatar. The user converses with the avatar, and their utterances and emotional information are captured by the device and sent to the server. The server analyzes the utterances and emotional information, generates appropriate responses, and sends them to the device. The device then provides the user with responses tailored to their emotions through the avatar.

[1586] Specific examples

[1587] When a user is walking near Shinjuku Station, the device acquires location information and sends it to the server. The server searches for recommended restaurant information and weather information around Shinjuku Station, generates optimal information based on the location information, and sends it to the device. The device then announces, "Here are some recommended restaurants," and displays the information on the AR screen. If the user shows signs of fatigue, the system also suggests nearby rest areas and coffee shops.

[1588] In addition, if a user is conversing in English with a foreign tourist, the device captures the conversation and sends it to the server. The server translates the English into Japanese and sends the translation result to the device. The device then displays the translated Japanese as subtitles, allowing the user to communicate smoothly. Furthermore, if the user is feeling tense, the system provides translation results using shorter sentences and simpler phrases.

[1589] As described above, the smart ARVR glasses system of the present invention integrates multiple functions, such as location information, voice capture, translation, video analysis, and emotion recognition, to provide users with a wide range of information support in real time. Furthermore, by incorporating an emotion engine, it is possible to flexibly respond to the user's emotional state, realizing a personalized user experience.

[1590] The processing flow will be explained below.

[1591] Information acquisition and provision process

[1592] Step 1:

[1593] Device: Uses the GPS sensor to obtain current location information.

[1594] Step 2:

[1595] Terminal: Sends the acquired location information to the server via the network communication module.

[1596] Step 3:

[1597] Server: Based on the received location information, the server searches a database for information such as nearby restaurants, tourist spots, dangerous roads, and weather forecasts.

[1598] Step 4:

[1599] Server: Optimizes search results and generates the information provided.

[1600] Step 5:

[1601] Server: Sends the generated information to the terminal.

[1602] Step 6:

[1603] Terminal: Provides received information to the user via voice or an AR screen.

[1604] Emotion Engine Functions

[1605] Step 1:

[1606] Device: Uses a microphone and camera to capture the user's voice and facial expressions.

[1607] Step 2:

[1608] Terminal: Sends captured audio and video data to the server.

[1609] Step 3:

[1610] Server: Recognizes user emotions using an emotion analysis engine.

[1611] Step 4:

[1612] Server: Adjust how information is presented based on perceived emotions.

[1613] Machine translation function

[1614] Step 1:

[1615] User: Start a conversation with someone who speaks a foreign language.

[1616] Step 2:

[1617] Device: Uses the microphone to capture audio.

[1618] Step 3:

[1619] Device: Sends captured audio data to the server.

[1620] Step 4:

[1621] Server: The voice data is passed through a translation engine and translated into the specified language.

[1622] Step 5:

[1623] Server: Adjusts translation results taking into account the user's emotional information.

[1624] Step 6:

[1625] Server: Sends the translation results to the device.

[1626] Step 7:

[1627] Device: Providing translated audio and subtitles to users.

[1628] Check road conditions in real time and avoid accidents

[1629] Step 1:

[1630] Device: Uses a 360-degree camera to capture footage of the surroundings in real time.

[1631] Step 2:

[1632] Terminal: Sends captured video data to the server.

[1633] Step 3:

[1634] Server: Uses a video analytics engine to detect dangerous situations and potential accidents.

[1635] Step 4:

[1636] Server: Adjust the warning content taking into account the user's emotional information.

[1637] Step 5:

[1638] Server: Sends the detection results to the device.

[1639] Step 6:

[1640] Device: Provides warning information to users via voice or AR screen.

[1641] Work support

[1642] Step 1:

[1643] User: Arrives at a designated work area on a manufacturing or logistics site.

[1644] Step 2:

[1645] Device: Obtains current location information and sends it to the server.

[1646] Step 3:

[1647] Server: Based on the location information, it searches the database for work procedures relevant to that area.

[1648] Step 4:

[1649] Server: Recognizes user emotions using an emotion analysis engine.

[1650] Step 5:

[1651] Server: Adjusts work procedures based on perceived emotions.

[1652] Step 6:

[1653] Server: Sends the adjusted work instructions to the device.

[1654] Step 7:

[1655] Terminal: Work procedures are displayed in AR in the user's field of view.

[1656] Customer service support

[1657] Step 1:

[1658] User: Starts a conversation with a customer.

[1659] Step 2:

[1660] Device: Use voice recognition to capture what you say.

[1661] Step 3:

[1662] Terminal: Sends captured conversation data to the server.

[1663] Step 4:

[1664] Server: Analyzes conversation data and generates optimal response examples.

[1665] Step 5:

[1666] Server: Recognizes user emotions using an emotion analysis engine.

[1667] Step 6:

[1668] Server: Adjusts response examples based on the recognized emotion.

[1669] Step 7:

[1670] Server: Sends the adjusted response examples to the device.

[1671] Step 8:

[1672] Device: Provide users with examples of what works with audio and subtitles.

[1673] English conversation lessons

[1674] Step 1:

[1675] User: Switches to VR mode to begin an English conversation lesson.

[1676] Step 2:

[1677] Terminal: Display the generated AI model avatar.

[1678] Step 3:

[1679] User: Converse with an AI avatar.

[1680] Step 4:

[1681] Device: Captures user speech and emotion information and sends it to the server.

[1682] Step 5:

[1683] Server: Analyzes the user's comments and emotional information and generates an appropriate response.

[1684] Step 6:

[1685] Server: Sends the adjusted response to the device.

[1686] Step 7:

[1687] Terminal: Provides tailored responses to users through avatars.

[1688] Example 2

[1689] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1690] Conventional information provision systems can only provide limited information based on the user's location information, making it difficult to respond flexibly to the user's emotions and situation. Furthermore, communication support in different languages ​​and detection and warning of dangerous situations can only respond uniformly without considering the user's emotions, so there is a need to improve the user experience.

[1691] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1692] In this invention, the server includes a means for acquiring location information, a means for searching and generating surrounding information based on the acquired location information, a means for displaying the generated information, and a means for recognizing the user's emotions and adjusting the information according to the emotions. This makes it possible to not only provide surrounding information based on the user's location information, but also to provide appropriate information, translation, and warnings according to the user's emotions.

[1693] "Location Information" means data of a user's current physical geographic coordinates obtained using a GPS sensor or other location detection device.

[1694] "Nearby information" refers to information related to the user's current location, such as nearby restaurants, tourist attractions, and weather information.

[1695] "Means for recognizing emotions" refers to technology that analyzes the user's voice, facial expressions, and movement data to estimate the user's emotional state.

[1696] A "360-degree camera" is a camera device capable of shooting in all directions, capturing a wide range of images at once.

[1697] A "dangerous situation" is a situation in the surroundings that may affect the user's safety, such as an accident or an obstacle.

[1698] "Adjusting according to emotions" means appropriately changing the information and warning content provided based on the user's current emotional state.

[1699] "Speech in a different language" refers to speech data in a language other than the language spoken by the user.

[1700] "Means for translating" refers to technology for converting audio data provided in a particular language into another language.

[1701] "Generated information" refers to data such as restaurant information and warnings generated based on location information, surrounding conditions, and the user's emotional state.

[1702] "Means of providing to the user" refers to the technology or device used to convey the generated information to the user in the form of audio, AR screen, text, etc.

[1703] The smart ARVR glasses system of the present invention is a multifunctional device that allows users to obtain various information in real time, enabling efficient communication and work support. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusting the information provided according to the user's emotions, it can provide a more personalized user experience.

[1704] System configuration

[1705] The system consists of the following main components:

[1706] 1. Device (Smart ARVR Glasses)

[1707] GPS sensor

[1708] 360-degree camera

[1709] microphone

[1710] speaker

[1711] Display (AR / VR mode switchable)

[1712] Network Communication Module

[1713] Emotion Engine

[1714] 2. Server

[1715] Database Management Systems

[1716] Translation Engine

[1717] Video Analysis Engine

[1718] Sentiment Analysis Engine

[1719] Various information services such as weather forecasts, restaurant information, and tourist information

[1720] Information acquisition and provision process

[1721] When a user puts on the smart ARVR glasses and begins using them, the device first obtains its current location using the GPS sensor. This location information is then sent to the server via the network communication module. The server then uses the received location information to search a database for information such as nearby restaurants, tourist spots, road hazards, and weather forecasts, and generates optimized information. The generated information is then sent to the device and provided to the user via voice or an AR screen.

[1722] Emotion Engine Functions

[1723] The emotion engine recognizes the user's current emotion from their voice, facial expressions, and actions, and uses the results to adjust the behavior of the entire system. For example, if the user is feeling stressed, the system will adjust to provide more concise and friendly information.

[1724] Machine translation function

[1725] When a user starts a conversation with someone speaking a foreign language, the device's microphone captures the voice, and the captured voice data is sent to the server along with emotional information. The server then passes the voice data and emotional information through a translation engine to translate the speech according to the user's emotions. The translation results are then sent to the device and provided to the user as audio or subtitles.

[1726] Check road conditions in real time and avoid accidents

[1727] While the user is moving, the device's 360-degree camera captures images of the surroundings in real time. This video data, along with emotional information, is sent to a server, which then uses a video analysis engine to detect dangerous situations and potential accidents. The detection results are then sent to the device, and a warning is displayed on the AR screen or via audio based on the user's emotional state.

[1728] Work support

[1729] When a user arrives at a designated work area at a manufacturing or logistics site, the device transmits location information and emotion information to a server. The server searches a database for relevant work procedures based on the location and emotion information, and transmits the retrieved work procedures to the device. The device then displays the procedures in AR, adjusted according to the user's emotion.

[1730] Customer service support

[1731] When a user starts a conversation with a customer while serving them, the device uses its voice recognition function to capture the conversation. The captured conversation data is sent to a server along with emotional information, and an analysis engine generates an optimal response example. The generated response example is sent to the device and provided to the user as audio or subtitles.

[1732] English conversation lessons

[1733] When a user starts an English conversation lesson, the device switches to VR mode and displays a generated AI model avatar. The user converses with the avatar, and their utterances and emotional information are captured by the device and sent to the server. The server analyzes the utterances and emotional information, generates appropriate responses, and sends them to the device. The device then provides the user with responses tailored to their emotions through the avatar.

[1734] Specific examples

[1735] For example, if a user is walking near Shinjuku Station, the device acquires location information and sends it to the server. The server searches for recommended restaurant information and weather information around Shinjuku Station, generates optimal information based on the location information, and sends it to the device. The device then announces, "Here are some recommended restaurants," and displays the information on the AR screen. If the user shows signs of fatigue, the system will also suggest nearby rest areas and coffee shops.

[1736] In addition, if a user is conversing in English with a foreign tourist, the device captures the conversation and sends it to the server. The server translates the English into Japanese and sends the translation result to the device. The device then displays the translated Japanese as subtitles, allowing the user to communicate smoothly. Furthermore, if the user is feeling tense, the system provides translation results using shorter sentences and simpler phrases.

[1737] Prompt Sentence Examples

[1738] An example prompt using a generative AI model would be:

[1739] "Please explain the process of providing information when a user walking near Shinjuku Station wants to check nearby restaurant recommendations. Also, please provide a specific example of how to respond if the user feels fatigued."

[1740] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1741] Step 1:

[1742] Acquiring and sending location information

[1743] The device uses the GPS sensor to obtain its current location.

[1744] Input: The user's current location.

[1745] Output: Real-time location data.

[1746] The acquired location information is transmitted to a server via a network communication module.

[1747] Operation: The device will announce "Current location obtained" and send data such as "Location information: Near Shinjuku Station" to the server.

[1748] Step 2:

[1749] Information Retrieval and Optimization

[1750] Based on the received location information, the server searches its database for nearby restaurants, tourist attractions, weather information, dangerous road information, and more.

[1751] Input: Location information sent from the device.

[1752] Output: Neighborhood information data as search results.

[1753] The server optimizes the acquired information and generates information to be presented to the user.

[1754] What it does: The server sends a query to the database to search for "recommended restaurants around Shinjuku Station." Based on the search results, it logs "Restaurant information generated."

[1755] Step 3:

[1756] Information provision and user emotional response

[1757] The device receives the generated information and provides it to the user via AR screen and audio.

[1758] Input: Optimized information data sent from the server.

[1759] Output: Visual and audio information displayed and notified to the user.

[1760] The device captures the user's reactions and emotions to the information provided.

[1761] How it works: The device will say "Here's a recommended restaurant" and display the restaurant's location on the AR screen. The device will capture the user's reaction and store it as emotion information.

[1762] Step 4:

[1763] Emotion Engine Operation

[1764] The server receives the user's voice, facial expression, and movement data and analyzes it using an emotion analysis engine.

[1765] Input: User voice, facial expression, and movement data sent from the device.

[1766] Output: Sentiment analysis result data.

[1767] The next information provided is adjusted according to the user's emotions.

[1768] Operation: If data is captured showing that the user is feeling tired, the emotion analysis engine detects "fatigue" and executes a process to "suggest nearby rest areas." Related information is sent to the device.

[1769] Step 5:

[1770] Machine translation function

[1771] The device uses a microphone to capture the user's voice and transmits it to the server along with emotional information.

[1772] Input: User's voice data and emotion information.

[1773] Output: Captured audio data and emotion information.

[1774] The server uses a translation engine to translate the captured audio.

[1775] Input: Voice data and emotion information sent from the device.

[1776] Output: Translation result data.

[1777] The system adjusts the translation based on the user's emotions and sends the translation results to the device.

[1778] Operation: The device notifies the user that "Audio data has been sent for translation," and the server logs "English to Japanese translation completed," and sends the translation results to the device.

[1779] Step 6:

[1780] Real-time road conditions and alerts

[1781] The device uses a 360-degree camera to capture images of the surroundings and send them to a server.

[1782] Input: Captured ambient video data and emotion information.

[1783] Output: Real-time video data.

[1784] The server uses a video analytics engine to detect dangerous situations.

[1785] Input: Video data sent from the device.

[1786] Output: Hazardous situation detection data.

[1787] The detection results are sent to the terminal and a warning is issued to the user.

[1788] What it does: The device displays "Capturing real-time video," the server logs "Dangerous intersection detected," and sends a warning to the device saying "Be careful."

[1789] Step 7:

[1790] Work support

[1791] When the user arrives at the work area, the device transmits location information and emotion information to the server.

[1792] Input: Location information obtained in the working area and user emotion information.

[1793] Output: Sending data to the server.

[1794] The server searches a database for relevant work procedures based on the location information and emotion information.

[1795] Input: Location and emotion information sent from the device.

[1796] Output: The searched work procedure data.

[1797] The acquired work procedures are sent to the terminal, and procedures adjusted according to the user's emotions are displayed in AR in the user's field of view.

[1798] Operation: The device issues a voice notification saying "Arrived at work area" and sends its location information to the server. The server logs "Work procedure generated" and sends it to the device.

[1799] Step 8:

[1800] Customer service support

[1801] The device uses voice recognition to capture conversations with customers.

[1802] Input: Customer conversation audio data.

[1803] Output: Captured conversation data and emotion information.

[1804] The server uses an analysis engine to generate the best possible response.

[1805] Input: Conversational data and emotional information sent from the device.

[1806] Output: Generated example data.

[1807] The generated examples are sent to the terminal and provided to the user as audio and subtitles.

[1808] How it works: The device displays "Capturing conversation," the server logs "Suggested response generated," and sends a suggestion to the device saying, "Here's what to say next."

[1809] Step 9:

[1810] English conversation lessons

[1811] The device switches to VR mode and displays the generated AI model avatar. The user's speech and emotional information are sent to the server.

[1812] Input: User utterance data and emotion information.

[1813] Output: Captured speech data and emotion information.

[1814] The server analyzes the content of the statement and emotional information and generates an appropriate response.

[1815] Input: Speech data and emotion information sent from the device.

[1816] output: The generated response data.

[1817] The generated response is sent to the terminal and provided to the user through the avatar.

[1818] Operation: The device announces "Start English conversation lesson" and displays an avatar. The server logs "Analyzing user's comment" and generates an appropriate response and sends it to the device.

[1819] The above are the specific processing steps of the program of this system.

[1820] (Application example 2)

[1821] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1822] Conventional ARVR systems and information acquisition systems provide information uniformly without considering the user's state or emotions, making it difficult to achieve a personalized user experience. While efficient real-time information provision is particularly required in logistics centers and factories, conventional systems have difficulty responding flexibly to the worker's emotions and situation. Furthermore, smooth communication between different languages ​​is difficult because translation results are provided without taking emotions into account. Furthermore, detection of dangerous situations and warnings during work do not reflect emotional information, making it difficult to respond appropriately to the user's state.

[1823] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring location information, means for displaying generated information, means for recognizing emotions, means for adjusting information to be provided according to the recognized emotions, means for capturing voices in different languages, means for providing translation results, means for translating the captured voices into a specified language, means for capturing surrounding video using a 360-degree camera, means for analyzing the captured video to detect dangerous situations, means for alerting the user of detected danger information, and means for acquiring emotional information and adjusting the content of the alert based on the analysis results. This makes it possible to provide personalized information based on the user's location information and emotional state and improve safety.

[1824] 1. "Means for obtaining location information" refers to a device that uses a GPS sensor or other location-determining technology to determine the user's current location.

[1825] 2. "Means for searching and generating surrounding information" refers to a processing system that searches a database for information about the surrounding environment and available services based on location information and provides it to the user.

[1826] 3. "Means for displaying generated information" refers to devices such as displays and speakers that provide acquired information to the user visually or audibly.

[1827] 4. "Means for recognizing emotions" refers to software and sensors for detecting and analyzing the user's emotional state from their voice, facial expressions, movements, etc.

[1828] 5. "Means for adjusting the information provided in response to the recognized emotion" means a processing system that changes the content or format of the information provided based on the detected emotion information.

[1829] 6. "Means for capturing audio in different languages" means microphones and speech analysis devices for collecting audio data corresponding to multiple languages.

[1830] 7. "Means for translating captured audio into a specified language" means translation software or services that convert captured audio data into another language.

[1831] 8. "Means for providing translation results" means a display or speaker for providing the translated audio or text to the user.

[1832] 9. "Means for capturing surrounding images using a 360-degree camera" means a camera device for capturing images in all directions.

[1833] 10. "Means for analyzing captured video to detect dangerous situations" means software that analyzes captured video data and recognizes dangerous situations and obstacles.

[1834] 11. "Means for warning the user of detected danger information" refers to a device that issues a visual or audible warning to the user based on the analysis results.

[1835] 12. "Means for acquiring emotional information and adjusting the content of warnings based on the analysis results" refers to a processing system for changing the content and format of warnings based on the user's emotional state.

[1836] Overall system overview

[1837] The system of this invention uses smart ARVR glasses to provide information and support to logistics center workers. This system acquires location information, analyzes emotional information, and provides the generated information to users, enabling efficient and safe work.

[1838] Hardware Configuration

[1839] 1. Device (Smart ARVR Glasses):

[1840] GPS sensor: Obtain location information

[1841] 360-degree camera: Capture footage of your surroundings

[1842] Microphone, speaker: Audio capture and output

[1843] Display (AR / VR mode switchable)

[1844] Network communication module: Data communication with the server

[1845] Emotion engine: Recognizes user emotions and adjusts information delivery accordingly

[1846] 2. Server:

[1847] Database Management System (DBMS): MariaDB

[1848] Translation engine: Google Cloud Translation API

[1849] Video analysis engine: OpenCV

[1850] Sentiment analysis engine: IBM Watson Tone Analyzer

[1851] Various information service APIs (work procedures, hazard information, etc.)

[1852] Software Configuration and Processing

[1853] The server processes information using the following means.

[1854] 1. Obtaining and sending location information:

[1855] The device's GPS sensor acquires location information and transmits it to the server via the network communication module. The server then references the database based on the location information to search for and generate relevant work procedures.

[1856] 2. Acquisition and analysis of emotional information:

[1857] The emotion engine captures the user's voice and facial expressions and sends them to the emotion analysis engine. The server analyzes the emotional information and generates optimal information according to the user's state.

[1858] 3. Video capture and analysis:

[1859] The 360-degree camera captures the surroundings and sends the video data to a server, which uses a video analytics engine to detect dangerous situations and alert the user.

[1860] Specific examples

[1861] 1. Display of work procedure:

[1862] When the user arrives at the designated work area, the device acquires the user's location information and sends it to the server. The server searches the database for relevant work procedures based on the location information, and sends the acquired work procedures to the device. The device then displays the procedures in AR, adjusted according to the user's emotions.

[1863] Example prompt: Tell me the procedure, and the location is "34.052235, -118.243683"

[1864] 2. Notification of dangerous situations:

[1865] While the user is moving, a 360-degree camera captures the surroundings and sends it to a server. The server uses a video analysis engine to detect dangerous situations and warn the user. Emotional information is also analyzed, and the warning content is adjusted accordingly.

[1866] Example prompt: Tells about danger around you and makes you feel nervous

[1867] 3. Multilingual Translation:

[1868] When a user starts a conversation with someone who speaks a different language, the device's microphone captures the voice and sends it along with emotional information to the server. The server then uses a translation engine to translate the voice into the specified language and sends the translation result to the device. The translation result is adjusted appropriately according to the user's emotional state.

[1869] Example prompt: Translate the conversation and the emotion is "stress"

[1870] As a result, the present invention enables workers in a logistics center to receive personalized support and perform their work safely and efficiently.

[1871] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1872] Program processing steps

[1873] Process flow for displaying work procedures

[1874] Step 1:

[1875] The device's GPS sensor acquires the user's current location. The input is location data from the GPS sensor, and the output is the acquired location information. Specifically, the GPS sensor periodically acquires location data and stores it in internal memory.

[1876] Step 2:

[1877] The location information acquired by the terminal is sent to the server via the network communication module. The input is location information, and the output is location data sent to the server. Specifically, the terminal converts the location information into packets and sends them to the server via the network communication module over the Internet.

[1878] Step 3:

[1879] The server refers to the database management system (DBMS) based on the location information it receives and searches for related work procedures. The input is the location information received by the server, and the output is the searched work procedure data. Specifically, the server generates an SQL query and retrieves related work procedures from the database.

[1880] Step 4:

[1881] The server sends the retrieved work procedure to the terminal via the network communication module. The input is the retrieved work procedure data, and the output is the work procedure to be sent to the terminal. Specifically, the server converts the work procedure data into packets and sends them to the terminal via the network communication module.

[1882] Step 5:

[1883] The device's emotion engine analyzes the user's voice and facial expressions to obtain emotional information. The input is voice data and facial expression data, and the output is emotional information. Specifically, the device captures voice and facial expressions using a microphone and camera, and processes them using the built-in emotion analysis engine.

[1884] Step 6:

[1885] The device adjusts the displayed work steps based on the emotional information it acquires. The input is emotional information and work step data, and the output is the adjusted work steps. Specifically, the emotion engine analyzes the emotional information and changes the display format and content to suit the user's state.

[1886] Step 7:

[1887] The device displays the adjusted work procedure in AR in the user's field of view. The input is the adjusted work procedure data, and the output is visual information provided to the user. Specifically, the device display displays the adjusted work procedure in AR mode, allowing the user to visually confirm it.

[1888] Processing flow for notification of dangerous situations

[1889] Step 1:

[1890] The device's 360-degree camera captures the surroundings. The input is the camera's video data, and the output is the captured video data. Specifically, the camera periodically captures video and stores it in its internal memory.

[1891] Step 2:

[1892] The video data captured by the terminal is sent to the server via the network communication module. The input is video data, and the output is video data sent to the server. Specifically, the terminal converts the video data into packets and sends them to the server via the network.

[1893] Step 3:

[1894] The video data received by the server is analyzed by a video analysis engine to detect dangerous situations. The input is video data and the output is detected danger information. Specifically, the video analysis engine analyzes the video data and automatically detects dangerous situations.

[1895] Step 4:

[1896] The server sends danger information to the terminal via the network communication module. The input is danger information, and the output is danger data sent to the terminal. Specifically, the server converts the danger information into packets and sends them to the terminal via the network communication module.

[1897] Step 5:

[1898] The device's emotion engine analyzes the user's voice and facial expressions to obtain emotional information. The input is voice data and facial expression data, and the output is emotional information. Specifically, the device captures voice and facial expressions using a microphone and camera, and processes them using the built-in emotion analysis engine.

[1899] Step 6:

[1900] The device adjusts the warning content based on danger information and emotional information. The input is danger information and emotional information, and the output is the adjusted warning content. Specifically, the emotion engine analyzes the emotional information and changes the content and format of the warning to suit the user's state.

[1901] Step 7:

[1902] The device notifies the user of the adjusted warning visually or audibly. The input is the adjusted warning content, and the output is the warning information provided to the user. Specifically, the device's display or speaker displays or notifies the user of the adjusted warning, allowing the user to respond immediately.

[1903] The above are the specific processing steps for carrying out the invention.

[1904] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1905] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1906] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1907] [Fourth embodiment]

[1908] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1909] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1910] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1911] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1912] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1913] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1914] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1915] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1916] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1917] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1918] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1919] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1920] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1921] The smart ARVR glasses system of the present invention is a multi-functional device that allows users to obtain various information in real time and provide efficient communication and work assistance.

[1922] System configuration

[1923] The system consists of the following main components:

[1924] 1. Device (Smart ARVR Glasses)

[1925] GPS sensor

[1926] 360-degree camera

[1927] microphone

[1928] speaker

[1929] Display (AR / VR mode switchable)

[1930] Network Communication Module

[1931] 2. Server

[1932] Database Management Systems

[1933] Translation Engine

[1934] Video Analysis Engine

[1935] Various information services such as weather forecasts, restaurant information, and tourist information

[1936] Information acquisition and provision process

[1937] When a user puts on the smart ARVR glasses and begins using them, the device first obtains its current location using the GPS sensor. This location information is then sent to the server via the network communication module. The server then uses the received location information to search its database for information on nearby restaurants, tourist spots, dangerous roads, weather forecasts, and other information, generating optimized information. The generated information is then sent to the device and provided to the user via voice or an AR screen.

[1938] Machine translation function

[1939] When a user starts a conversation with someone who speaks a foreign language, the device's microphone captures the voice and sends the captured voice data to the server. The server then passes the voice data through a translation engine to translate it into the required language. The translation result is then sent to the device and provided to the user as audio or subtitles.

[1940] Check road conditions in real time and avoid accidents

[1941] While the user is moving, the device's 360-degree camera captures video of the surroundings in real time. This video data is sent to a server, which uses a video analytics engine to detect dangerous situations or potential accidents. The results are then sent to the device, and a warning is displayed to the user via audio and an AR screen.

[1942] Work support

[1943] When a user arrives at a designated work area at a manufacturing or logistics site, the device sends location information to a server. The server searches a database for work procedures related to that area based on the location information, and sends the retrieved work procedures to the device. The device then displays the procedures in AR within the user's field of view, helping to improve work efficiency.

[1944] Customer service support

[1945] When a user starts a conversation with a customer while serving them, the device uses its voice recognition function to capture the conversation. The captured conversation data is sent to a server, where an analysis engine generates the most appropriate response. The generated response is sent to the device and provided to the user as audio or subtitles.

[1946] English conversation lessons

[1947] When a user starts an English conversation lesson, the device switches to VR mode and displays a generated AI model avatar. The user converses with the avatar, and their utterances are captured by the device and sent to the server. The server analyzes the utterances, generates appropriate responses, and sends them to the device. The device then provides the generated responses to the user through the avatar.

[1948] Specific examples

[1949] When a user is walking near Shinjuku Station, the device acquires location information and sends it to the server. The server searches for recommended restaurants and weather information around Shinjuku Station, generates the most appropriate information, and sends it to the device. The device then announces, "Here are some recommended restaurants," and displays it on the AR screen.

[1950] In addition, if a user is conversing in English with a foreign tourist, the device captures the conversation and sends it to the server. The server translates the English into Japanese and sends the translation result to the device. The device displays the translated Japanese as subtitles, allowing the user to communicate smoothly.

[1951] As described above, the smart ARVR glasses system of the present invention can provide users with a wide range of information support in real time by integrating multiple functions such as location information, voice capture, translation, and video analysis.

[1952] The processing flow will be explained below.

[1953] Information acquisition and provision process

[1954] Step 1:

[1955] Device: Uses the GPS sensor to obtain current location information.

[1956] Step 2:

[1957] Terminal: Sends the acquired location information to the server via the network communication module.

[1958] Step 3:

[1959] Server: Based on the received location information, the server searches a database for information such as nearby restaurants, tourist spots, dangerous roads, and weather forecasts.

[1960] Step 4:

[1961] Server: Optimizes search results and generates the information provided.

[1962] Step 5:

[1963] Server: Sends the generated information to the terminal.

[1964] Step 6:

[1965] Terminal: Provides received information to the user via voice or an AR screen.

[1966] Machine translation function

[1967] Step 1:

[1968] User: Start a conversation with someone who speaks a foreign language.

[1969] Step 2:

[1970] Device: Uses the microphone to capture audio.

[1971] Step 3:

[1972] Device: Sends captured audio data to the server.

[1973] Step 4:

[1974] Server: The received voice data is passed through a translation engine and translated into the specified language.

[1975] Step 5:

[1976] Server: Sends the translation results to the device.

[1977] Step 6:

[1978] Device: Providing translated audio and subtitles to users.

[1979] Check road conditions in real time and avoid accidents

[1980] Step 1:

[1981] Device: Uses a 360-degree camera to capture footage of the surroundings in real time.

[1982] Step 2:

[1983] Terminal: Sends captured video data to the server.

[1984] Step 3:

[1985] Server: Analyzes the received video data and detects dangerous situations and potential accidents.

[1986] Step 4:

[1987] Server: Sends the detection results to the device.

[1988] Step 5:

[1989] Device: Provides warning information to users via voice or AR screen.

[1990] Work support

[1991] Step 1:

[1992] User: Arrives at a designated work area on a manufacturing or logistics site.

[1993] Step 2:

[1994] Device: Obtains current location information and sends it to the server.

[1995] Step 3:

[1996] Server: Based on the location information, it searches the database for work procedures relevant to that area.

[1997] Step 4:

[1998] Server: Sends search results to the device.

[1999] Step 5:

[2000] Terminal: Work procedures are displayed in AR in the user's field of view.

[2001] Customer service support

[2002] Step 1:

[2003] User: Starts a conversation with a customer.

[2004] Step 2:

[2005] Device: Use voice recognition to capture what you say.

[2006] Step 3:

[2007] Terminal: Sends captured conversation data to the server.

[2008] Step 4:

[2009] Server: Analyzes conversation data and generates optimal response examples.

[2010] Step 5:

[2011] Server: Sends the generated response examples to the device.

[2012] Step 6:

[2013] Device: Provide users with examples of what works with audio and subtitles.

[2014] English conversation lessons

[2015] Step 1:

[2016] User: Switches to VR mode to begin an English conversation lesson.

[2017] Step 2:

[2018] Terminal: Display the generated AI model avatar.

[2019] Step 3:

[2020] User: Converse with an AI avatar.

[2021] Step 4:

[2022] Terminal: Captures what the user says and sends it to the server.

[2023] Step 5:

[2024] Server: Analyzes what the user says and generates an appropriate response.

[2025] Step 6:

[2026] Server: Generates a response and sends it to the device.

[2027] Step 7:

[2028] Terminal: The generated response is presented to the user through an avatar.

[2029] Example 1

[2030] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2031] In conventional smart devices and information systems, functions such as location-based information provision, language translation, and hazard detection were implemented separately, forcing users to use multiple devices and applications simultaneously. Furthermore, because the various functions were not linked in real time, it was difficult to efficiently obtain information or provide work support. Therefore, a system that integrated all functions was needed.

[2032] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2033] In this invention, the server includes means for acquiring location information, means for searching for surrounding information based on the acquired location information, means for generating optimal information based on the searched information, means for displaying the generated information, means for capturing audio in different languages, means for translating the captured audio into a specified language, means for providing the translated result, means for capturing video, means for analyzing the captured video to detect a dangerous situation, means for alerting the user of detected danger information, means for inputting a prompt sentence, means for analyzing the input prompt sentence, means for generating an appropriate response based on the analysis result, and means for providing the generated response. This enables a single system to provide multiple functions including information provision, translation, danger detection, and conversation support based on prompts.

[2034] "Means for acquiring location information" refers to a device or system for measuring the current location and acquiring that information as digital data.

[2035] The "means for searching surrounding information" is a system for extracting information about the region and environment from a database based on the acquired location information.

[2036] "Means for generating optimal information" refers to algorithms or programs that evaluate the searched information, convert it into the most useful form for the user, and provide it to the user.

[2037] The "means for displaying information" refers to a device or interface for visually or audibly presenting the generated information to the user.

[2038] A "means for capturing audio in different languages" is a system for using a device such as a microphone to pick up audio spoken in different languages ​​and record it as digital data.

[2039] A "means for translating speech into a specified language" is software or algorithms that analyze the captured speech data and translate it into a target language.

[2040] "Means for providing translated results" refers to a function or device for outputting translated language data to the user in the form of voice or text.

[2041] A "means for capturing video" is a system that uses a device such as a camera to capture video of the surroundings and record it as digital data.

[2042] "Means for analyzing captured video to detect dangerous situations" refers to algorithms or programs that analyze the acquired video data and recognize dangerous situations or abnormal movements from it.

[2043] The "means for warning the user about danger information" refers to means such as sound, visual display, vibration, etc. for notifying the user of the detected danger information.

[2044] A "means for inputting prompt sentences" is an interface that allows the user to freely enter questions or instructions into the system through text input or voice input.

[2045] The "means for analyzing the input prompt sentence" is a natural language processing algorithm for understanding the prompt sentence input by the user and performing appropriate processing according to its content.

[2046] The "means for generating an appropriate response" is software or a model for generating an accurate and appropriate response to the user based on the analyzed prompt sentence.

[2047] The "means for providing a response" refers to a function or device that outputs the generated response to the user in voice or text form and supports the dialogue with the user.

[2048] The system of the present invention is an integrated information provision system that combines a smart device and a server. This makes it possible to provide multiple functions such as location information, translation, road condition confirmation, and prompt sentence analysis in an integrated manner. The specific configuration and operation procedure are described below.

[2049] System configuration

[2050] The system mainly consists of the following components:

[2051] 1. Terminal (smart device such as smart glasses)

[2052] GPS sensor

[2053] Cameras (including 360-degree cameras)

[2054] microphone

[2055] speaker

[2056] Display (AR / VR mode switchable)

[2057] Network communication module (Wi-Fi and mobile data communication)

[2058] 2. Server

[2059] Database Management Systems

[2060] Translation engine (e.g. Google Translate API)

[2061] Video Analysis Engine

[2062] information search engine

[2063] Generative AI Models

[2064] Main functions and processing flow

[2065] Information provision function

[2066] When a user puts on a smart device and starts using it, the device's GPS sensor acquires its current location. The acquired location information is sent to a server via a network communication module. Based on this location information, the server uses a database management system to search for information about the surrounding area and generate the most appropriate information from the search results. This information is then provided to the user in the form of audio, AR display, or other formats.

[2067] Specific examples

[2068] When a user is walking near Shinjuku Station, the device's GPS sensor acquires location information and sends it to the server. The server searches for recommended restaurants and weather information around Shinjuku Station, generates the most appropriate information, and sends it to the device. The device then announces, "Here are some recommended restaurants," and displays it on the AR screen.

[2069] Machine translation function

[2070] When a user starts a conversation with someone speaking a foreign language, the device's microphone captures the audio and sends it to the server. The server then translates the audio data into the required language through a translation engine and sends the translation results to the device. The translation results are then provided to the user as audio or subtitles.

[2071] Specific examples

[2072] When a user is conversing in English with a foreign tourist, the device's microphone captures the conversation and sends it to the server. The server then passes the voice data through a translation engine to translate the English into Japanese and sends the result to the device. The device then displays the translated Japanese as subtitles, allowing the user to communicate smoothly.

[2073] Road condition check function

[2074] While the user is moving, the device's 360-degree camera captures video of the surroundings. This video data is sent to a server, which uses a video analysis engine to detect dangerous situations or potential accidents. The results are then sent to the device, where they are used to warn the user via audio and AR displays.

[2075] Prompt sentence analysis function

[2076] When a user enters a prompt, the device sends the text or voice data to the server, which uses a generative AI model to analyze the prompt and generate an appropriate response, which is then sent to the device and provided to the user as voice or text.

[2077] Prompt Sentence Examples

[2078] "Please tell me some recommended restaurants near Shinjuku Station."

[2079] As described above, the system of the present invention provides multiple functions, including location information, voice capture, translation, video analysis, and prompt sentence analysis, in an integrated manner, and can provide multifunctional information support to users in real time.

[2080] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2081] Step 1:

[2082] The user wears a smart device and starts the system. The device acquires the current location information using the GPS sensor. This location information (input) is stored on the device as coordinate data (output).

[2083] Step 2:

[2084] The device sends the acquired location information to the server via the network communication module (input). At this time, the data is encrypted before being sent (specific operation).

[2085] Step 3:

[2086] The server uses the database management system to search for surrounding information based on the received location information (input), and extracts information about restaurants, tourist spots, dangerous road conditions, and other information from the database near the current location (output).

[2087] Step 4:

[2088] The server generates the most appropriate information for the user based on the search results (input). For example, if the user is a pedestrian, it will prioritize information about restaurants and weather within walking distance (output).

[2089] Step 5:

[2090] The server sends the generated optimized information to the terminal (input). The sent information includes audio data, text data, and video data (output).

[2091] Step 6:

[2092] The device then provides the received information to the user via voice or an AR screen. For example, it may say, "Here are some recommended restaurants," and display a map on the AR screen (specifically, an example of the area around Shinjuku Station).

[2093] Step 7:

[2094] When a user starts a conversation with someone who speaks a foreign language, the device's microphone captures the conversation (input) and sends the audio data to a server (output).

[2095] Step 8:

[2096] The server translates the received voice data using a translation engine (input). For example, to translate from English to Japanese, it uses the Google Translate API (output).

[2097] Step 9:

[2098] The server generates the translation result and sends it to the terminal (input), which contains the translated text data (output).

[2099] Step 10:

[2100] The device provides the translation results to the user as audio or subtitles, for example, by displaying them as subtitles to enable the user to converse smoothly (specific operation).

[2101] Step 11:

[2102] When the user inputs a prompt sentence, the terminal transmits the text data or voice data to the server (input).

[2103] Step 12:

[2104] The server uses a generative AI model to parse the prompt (input) and generate an appropriate response (output).

[2105] Step 13:

[2106] The server generates a response and sends it to the terminal (input), which may contain, for example, audio or text data (output).

[2107] Step 14:

[2108] The device then provides the generated response to the user via voice or text. For example, in response to the prompt, "Please tell me some recommended restaurants near Shinjuku Station," the device will provide information about recommended restaurants via voice (specific action).

[2109] (Application example 1)

[2110] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2111] In logistics centers, workers often face communication barriers when they need to efficiently receive work instructions over a wide area or when they need to use a foreign language. In addition, there is a need for real-time detection and response to work instructions and dangerous situations within large-scale facilities. These challenges must be addressed.

[2112] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2113] In this invention, the server includes a means for acquiring location information, a means for searching for and generating information about the surrounding area based on the acquired location information, a means for displaying the generated information, and a means for acquiring and displaying work instructions. This enables workers at a logistics center to efficiently receive instructions and perform appropriate work. Furthermore, a system including a means for capturing audio in different languages, a means for translating the captured audio into a specified language, a means for providing the translated results, and a means for displaying the translated results as support information facilitates communication between workers. Furthermore, a system including a means for capturing video of the surrounding area using a 360-degree camera, a means for analyzing the captured video to detect dangerous situations, a means for alerting users of detected danger information, and a means for improving work efficiency at logistics facilities enables improved safety and optimized work efficiency.

[2114] A "means for acquiring location information" is a device, equipment, or software that uses a GPS sensor or other location detection technology to identify and collect information about a person's current location.

[2115] "Means for searching and generating surrounding information" refers to a program or device that searches a database for information about the relevant area or location based on the acquired location information and generates the necessary information.

[2116] "Means for displaying generated information" refers to devices such as displays and AR / VR equipment that visually provide workers or users with the collected and generated information, or software for such devices.

[2117] "Means for obtaining and displaying work instructions" refers to devices or software that include a display or visual presentation method for obtaining specific work instructions from a server or other information source and displaying them in an easy-to-understand manner to the user.

[2118] The "means for capturing audio in different languages" is a microphone and audio collection software for collecting audio data in the languages ​​spoken by the user.

[2119] The "means for translating the captured voice into a specified language" is a translation engine or translation program for converting the captured voice data into another language.

[2120] "Means for providing translated results" means a display or audio player, or software for presenting the translated text or audio to the user.

[2121] The "means for displaying the translation results as support information" refers to a visual presentation device or interface for providing the translated text or audio to the user as work instructions or support information.

[2122] "Means for capturing surrounding images using a 360-degree camera" refers to a 360-degree camera for collecting surrounding images in all directions or an image collection device for such a purpose.

[2123] The "means for analyzing captured video to detect dangerous situations" refers to image analysis software or a program for analyzing acquired video data and detecting dangerous situations or abnormalities.

[2124] "Means for alerting the user to detected danger information" refers to a warning display device or audio alert device, or software for such devices, that notifies the user of detected danger or abnormality.

[2125] "Means for improving work efficiency at logistics facilities" refers to devices and software that provide users with appropriate work instructions and support information to make work within logistics facilities more efficient and smooth the flow of work.

[2126] MODE FOR CARRYING OUT THE INVENTION

[2127] A specific embodiment of the present invention will be described below. This embodiment is a system that uses smart glasses to improve work efficiency in a logistics center.

[2128] Overall system overview

[2129] The system consists of the following main components:

[2130] 1. Smart glasses (terminal)

[2131] 2. Server (backend system)

[2132] 3. Network Communication Infrastructure

[2133] Smart Glasses Configuration

[2134] The smart glasses have the following features:

[2135] GPS sensor: A sensor for obtaining location information.

[2136] 360-degree camera: A camera for capturing images of the surroundings.

[2137] Microphone: A device for capturing sound.

[2138] Speaker: A device for outputting sound.

[2139] Display: A display that can be displayed in AR / VR mode.

[2140] Network communication module: A module for communicating with the server.

[2141] Server Configuration

[2142] The server has the following features:

[2143] Database management system: Manages location-based surrounding information and work instructions.

[2144] Translation engine: Translates audio between different languages.

[2145] Video analysis engine: Analyzes video captured by a 360-degree camera to detect dangerous situations.

[2146] Information services: Providing necessary information such as weather forecasts and logistics information.

[2147] Information acquisition and provision process

[2148] 1. Obtaining location information

[2149] The GPS sensor in the smart glasses captures current location information and sends it to a server.

[2150] 2. Information retrieval and generation

[2151] Based on the received location information, the server searches the database for work instructions within the logistics center and generates optimized information.

[2152] 3. Displaying Information

[2153] Work instructions and surrounding information are displayed in AR on the smart glasses display and provided to workers, while information is also provided via audio.

[2154] 4. Voice translation function

[2155] When a worker initiates a conversation with a colleague who speaks a foreign language, the smart glasses' microphone captures the audio and sends it to the server, where the translation engine translates the speech and sends the results back to the smart glasses, where the translation results are displayed as subtitles on the display.

[2156] 5. Detecting dangerous situations

[2157] The 360-degree camera captures the surroundings in real time and sends it to a server. The server uses a video analytics engine to detect dangerous situations and sends that information to the smart glasses, which then provide a warning via AR display and audio.

[2158] Specific examples

[2159] For example, if a user working in a logistics center wants to know the location of a specific shelf, the smart glasses will obtain the location information and send it to the server. The server will then retrieve the location information from the database and display it on the user's glasses in AR. Furthermore, when the same user communicates with a worker who speaks a foreign language, the glasses can capture their voice and display the translation results on the display, enabling smooth conversation.

[2160] Example prompts to input to a generative AI model:

[2161] "You work at a logistics center in Tokyo. Please use the information below to navigate to the appropriate shelves and communicate with the foreign workers. Your current location is 35.6895, 139.6917."

[2162] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2163] Step 1:

[2164] The device's GPS sensor acquires the current location information. The input is location information, and the output is the acquired location data. The location data acquired from the GPS sensor is sent to the server.

[2165] Step 2:

[2166] The server analyzes the received location data and searches the database for work instructions within the logistics center based on that location information. The input is location data, and the output is the corresponding work instruction data. The database management system is used to generate the optimal work instructions.

[2167] Step 3:

[2168] The server sends the generated work instructions to the terminal. The terminal's display provides the work instructions to the user in an AR display. The input is the work instruction data, and the output is the AR display and audio instructions. The display and speaker are used to provide information visually and audibly.

[2169] Step 4:

[2170] The device's microphone captures audio in different languages. The input is audio data, and the output is the captured audio file. The audio data is sent to the server.

[2171] Step 5:

[2172] The server's translation engine analyzes the captured voice data and translates it into the specified language. The input is the voice data, and the output is the translated text data. The translation engine is used to generate an accurate translation.

[2173] Step 6:

[2174] The server sends the translation results to the terminal, and the terminal's display displays the translation results as subtitles. The input is the translated text data, and the output is the subtitle display. Visual information is provided to the user through the display.

[2175] Step 7:

[2176] The device's 360-degree camera captures the surroundings. The input is the video data, and the output is the captured video file. The video data is sent to the server.

[2177] Step 8:

[2178] The server's video analysis engine analyzes the captured video data and detects dangerous situations. The input is the video data and the output is the detected danger information. The video analysis engine is used to identify dangerous situations.

[2179] Step 9:

[2180] The server sends the detected danger information to the terminal, and the terminal's display and speaker provide a warning to the user. The input is the detected danger information, and the output is a warning display and audio warning. The warning is conveyed to the user using the display and speaker.

[2181] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2182] The smart ARVR glasses system of the present invention is a multifunctional device that allows users to obtain various information in real time, enabling efficient communication and work support. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusting the information provided according to the user's emotions, it can provide a more personalized user experience.

[2183] System configuration

[2184] The system consists of the following main components:

[2185] 1. Device (Smart ARVR Glasses)

[2186] GPS sensor

[2187] 360-degree camera

[2188] microphone

[2189] speaker

[2190] Display (AR / VR mode switchable)

[2191] Network Communication Module

[2192] Emotion Engine

[2193] 2. Server

[2194] Database Management Systems

[2195] Translation Engine

[2196] Video Analysis Engine

[2197] Sentiment Analysis Engine

[2198] Various information services such as weather forecasts, restaurant information, and tourist information

[2199] Information acquisition and provision process

[2200] When a user puts on the smart ARVR glasses and begins using them, the device first obtains its current location using the GPS sensor. This location information is then sent to the server via the network communication module. The server then uses the received location information to search its database for information on nearby restaurants, tourist spots, dangerous roads, weather forecasts, and other information, generating optimized information. The generated information is then sent to the device and provided to the user via voice or an AR screen.

[2201] Emotion Engine Functions

[2202] The emotion engine recognizes the user's current emotion from their voice, facial expressions, and actions, and uses the results to adjust the behavior of the entire system. For example, if the user is feeling stressed, the system will adjust to provide more concise and friendly information.

[2203] Machine translation function

[2204] When a user starts a conversation with someone speaking a foreign language, the device's microphone captures the voice, and the captured voice data is sent to the server along with emotional information. The server then passes the voice data and emotional information through a translation engine to translate the speech according to the user's emotions. The translation results are then sent to the device and provided to the user as audio or subtitles.

[2205] Check road conditions in real time and avoid accidents

[2206] While the user is moving, the device's 360-degree camera captures images of the surroundings in real time. This video data, along with emotional information, is sent to a server, which then uses a video analysis engine to detect dangerous situations and potential accidents. The detection results are then sent to the device, and a warning is displayed on the AR screen or via audio based on the user's emotional state.

[2207] Work support

[2208] When a user arrives at a designated work area at a manufacturing or logistics site, the device transmits location information and emotion information to a server. The server searches a database for relevant work procedures based on the location and emotion information, and transmits the retrieved work procedures to the device. The device then displays the procedures in AR, adjusted according to the user's emotion.

[2209] Customer service support

[2210] When a user starts a conversation with a customer while serving them, the device uses its voice recognition function to capture the conversation. The captured conversation data is sent to a server along with emotional information, and an analysis engine generates an optimal response example. The generated response example is sent to the device and provided to the user as audio or subtitles.

[2211] English conversation lessons

[2212] When a user starts an English conversation lesson, the device switches to VR mode and displays a generated AI model avatar. The user converses with the avatar, and their utterances and emotional information are captured by the device and sent to the server. The server analyzes the utterances and emotional information, generates appropriate responses, and sends them to the device. The device then provides the user with responses tailored to their emotions through the avatar.

[2213] Specific examples

[2214] When a user is walking near Shinjuku Station, the device acquires location information and sends it to the server. The server searches for recommended restaurant information and weather information around Shinjuku Station, generates optimal information based on the location information, and sends it to the device. The device then announces, "Here are some recommended restaurants," and displays the information on the AR screen. If the user shows signs of fatigue, the system also suggests nearby rest areas and coffee shops.

[2215] In addition, if a user is conversing in English with a foreign tourist, the device captures the conversation and sends it to the server. The server translates the English into Japanese and sends the translation result to the device. The device then displays the translated Japanese as subtitles, allowing the user to communicate smoothly. Furthermore, if the user is feeling tense, the system provides translation results using shorter sentences and simpler phrases.

[2216] As described above, the smart ARVR glasses system of the present invention integrates multiple functions, such as location information, voice capture, translation, video analysis, and emotion recognition, to provide users with a wide range of information support in real time. Furthermore, by incorporating an emotion engine, it is possible to flexibly respond to the user's emotional state, realizing a personalized user experience.

[2217] The processing flow will be explained below.

[2218] Information acquisition and provision process

[2219] Step 1:

[2220] Device: Uses the GPS sensor to obtain current location information.

[2221] Step 2:

[2222] Terminal: Sends the acquired location information to the server via the network communication module.

[2223] Step 3:

[2224] Server: Based on the received location information, the server searches a database for information such as nearby restaurants, tourist spots, dangerous roads, and weather forecasts.

[2225] Step 4:

[2226] Server: Optimizes search results and generates the information provided.

[2227] Step 5:

[2228] Server: Sends the generated information to the terminal.

[2229] Step 6:

[2230] Terminal: Provides received information to the user via voice or an AR screen.

[2231] Emotion Engine Functions

[2232] Step 1:

[2233] Device: Uses a microphone and camera to capture the user's voice and facial expressions.

[2234] Step 2:

[2235] Terminal: Sends captured audio and video data to the server.

[2236] Step 3:

[2237] Server: Recognizes user emotions using an emotion analysis engine.

[2238] Step 4:

[2239] Server: Adjust how information is presented based on perceived emotions.

[2240] Machine translation function

[2241] Step 1:

[2242] User: Start a conversation with someone who speaks a foreign language.

[2243] Step 2:

[2244] Device: Uses the microphone to capture audio.

[2245] Step 3:

[2246] Device: Sends captured audio data to the server.

[2247] Step 4:

[2248] Server: The voice data is passed through a translation engine and translated into the specified language.

[2249] Step 5:

[2250] Server: Adjusts translation results taking into account the user's emotional information.

[2251] Step 6:

[2252] Server: Sends the translation results to the device.

[2253] Step 7:

[2254] Device: Providing translated audio and subtitles to users.

[2255] Check road conditions in real time and avoid accidents

[2256] Step 1:

[2257] Device: Uses a 360-degree camera to capture footage of the surroundings in real time.

[2258] Step 2:

[2259] Terminal: Sends captured video data to the server.

[2260] Step 3:

[2261] Server: Uses a video analytics engine to detect dangerous situations and potential accidents.

[2262] Step 4:

[2263] Server: Adjust the warning content taking into account the user's emotional information.

[2264] Step 5:

[2265] Server: Sends the detection results to the device.

[2266] Step 6:

[2267] Device: Provides warning information to users via voice or AR screen.

[2268] Work support

[2269] Step 1:

[2270] User: Arrives at a designated work area on a manufacturing or logistics site.

[2271] Step 2:

[2272] Device: Obtains current location information and sends it to the server.

[2273] Step 3:

[2274] Server: Based on the location information, it searches the database for work procedures relevant to that area.

[2275] Step 4:

[2276] Server: Recognizes user emotions using an emotion analysis engine.

[2277] Step 5:

[2278] Server: Adjusts work procedures based on perceived emotions.

[2279] Step 6:

[2280] Server: Sends the adjusted work instructions to the device.

[2281] Step 7:

[2282] Terminal: Work procedures are displayed in AR in the user's field of view.

[2283] Customer service support

[2284] Step 1:

[2285] User: Starts a conversation with a customer.

[2286] Step 2:

[2287] Device: Use voice recognition to capture what you say.

[2288] Step 3:

[2289] Terminal: Sends captured conversation data to the server.

[2290] Step 4:

[2291] Server: Analyzes conversation data and generates optimal response examples.

[2292] Step 5:

[2293] Server: Recognizes user emotions using an emotion analysis engine.

[2294] Step 6:

[2295] Server: Adjusts response examples based on the recognized emotion.

[2296] Step 7:

[2297] Server: Sends the adjusted response examples to the device.

[2298] Step 8:

[2299] Device: Provide users with examples of what works with audio and subtitles.

[2300] English conversation lessons

[2301] Step 1:

[2302] User: Switches to VR mode to begin an English conversation lesson.

[2303] Step 2:

[2304] Terminal: Display the generated AI model avatar.

[2305] Step 3:

[2306] User: Converse with an AI avatar.

[2307] Step 4:

[2308] Device: Captures user speech and emotion information and sends it to the server.

[2309] Step 5:

[2310] Server: Analyzes the user's comments and emotional information and generates an appropriate response.

[2311] Step 6:

[2312] Server: Sends the adjusted response to the device.

[2313] Step 7:

[2314] Terminal: Provides tailored responses to users through avatars.

[2315] Example 2

[2316] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2317] Conventional information provision systems can only provide limited information based on the user's location information, making it difficult to respond flexibly to the user's emotions and situation. Furthermore, communication support in different languages ​​and detection and warning of dangerous situations can only respond uniformly without considering the user's emotions, so there is a need to improve the user experience.

[2318] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2319] In this invention, the server includes a means for acquiring location information, a means for searching and generating surrounding information based on the acquired location information, a means for displaying the generated information, and a means for recognizing the user's emotions and adjusting the information according to the emotions. This makes it possible to not only provide surrounding information based on the user's location information, but also to provide appropriate information, translation, and warnings according to the user's emotions.

[2320] "Location Information" means data of a user's current physical geographic coordinates obtained using a GPS sensor or other location detection device.

[2321] "Nearby information" refers to information related to the user's current location, such as nearby restaurants, tourist attractions, and weather information.

[2322] "Means for recognizing emotions" refers to technology that analyzes the user's voice, facial expressions, and movement data to estimate the user's emotional state.

[2323] A "360-degree camera" is a camera device capable of shooting in all directions, capturing a wide range of images at once.

[2324] A "dangerous situation" is a situation in the surroundings that may affect the user's safety, such as an accident or an obstacle.

[2325] "Adjusting according to emotions" means appropriately changing the information and warning content provided based on the user's current emotional state.

[2326] "Speech in a different language" refers to speech data in a language other than the language spoken by the user.

[2327] "Means for translating" refers to technology for converting audio data provided in a particular language into another language.

[2328] "Generated information" refers to data such as restaurant information and warnings generated based on location information, surrounding conditions, and the user's emotional state.

[2329] "Means of providing to the user" refers to the technology or device used to convey the generated information to the user in the form of audio, AR screen, text, etc.

[2330] The smart ARVR glasses system of the present invention is a multifunctional device that allows users to obtain various information in real time, enabling efficient communication and work support. Furthermore, by incorporating an emotion engine that recognizes the user's emotions and adjusting the information provided according to the user's emotions, it can provide a more personalized user experience.

[2331] System configuration

[2332] The system consists of the following main components:

[2333] 1. Device (Smart ARVR Glasses)

[2334] GPS sensor

[2335] 360-degree camera

[2336] microphone

[2337] speaker

[2338] Display (AR / VR mode switchable)

[2339] Network Communication Module

[2340] Emotion Engine

[2341] 2. Server

[2342] Database Management Systems

[2343] Translation Engine

[2344] Video Analysis Engine

[2345] Sentiment Analysis Engine

[2346] Various information services such as weather forecasts, restaurant information, and tourist information

[2347] Information acquisition and provision process

[2348] When a user puts on the smart ARVR glasses and begins using them, the device first obtains its current location using the GPS sensor. This location information is then sent to the server via the network communication module. The server then uses the received location information to search a database for information such as nearby restaurants, tourist spots, road hazards, and weather forecasts, and generates optimized information. The generated information is then sent to the device and provided to the user via voice or an AR screen.

[2349] Emo...

Claims

1. A means for acquiring location information; A means for searching and generating surrounding information based on the acquired location information; means for displaying the generated information; A system including:

2. a means for capturing audio in different languages; A means for translating the captured audio into a specified language; The system of claim 1 further comprising means for providing a translated result.

3. A means of capturing footage of your surroundings using a 360-degree camera; A means for analyzing the captured video to detect dangerous situations; 10. The system of claim 1, further comprising means for alerting a user to detected hazard information.

4. A means for acquiring location information of the work; The system according to claim 1, further comprising means for searching for and displaying related work procedures based on the acquired location information.

5. A means of capturing customer conversations and A means of analyzing the captured conversation and generating optimal response examples; The system of claim 1 further comprising means for providing the generated correspondence examples.

6. means having a virtual reality mode for conducting English conversation lessons; a means for capturing and analyzing a conversation between a user and a virtual avatar; The system of claim 1 further comprising means for generating and providing a virtual avatar response based on the analysis results.

Images