system

Abstract

Provide a system. 【Solution means】 An image acquisition means for capturing game information in real time, An analysis means for analyzing the game situation by analyzing the image, A voice guide generation means for generating a voice instruction based on the analysis result, A feedback generation means for generating a tactile feedback based on the analysis result, An output means for outputting the voice instruction and the tactile feedback, A voice input analysis means for acquiring and analyzing a user's voice input, An operation control means for controlling a user's game operation in response to the voice input, A system including the above.

Description

Technical Field

[0001] The technology of this disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a problem that visually impaired people have difficulty enjoying games as much as ordinary healthy people because they cannot access visual information. Also, since the screen situation of games constantly changes, appropriate accessibility needs to be provided each time, but there is a lack of support technologies to meet this requirement. As a result, there is a problem that the game experience that visually impaired people can participate in is limited.

Means for Solving the Problems

[0005] To solve the above-mentioned problems, the present invention provides an image acquisition means and an analysis means for capturing and analyzing game information in real time, and an audio guide generation means and a feedback generation means for generating audio instructions and haptic feedback according to the game situation. Furthermore, by combining an audio input analysis means for acquiring and analyzing user voice input and an operation control means for controlling game operations based on that voice input, the present invention provides a system that enables visually independent gameplay.

[0006] "Image acquisition means" refers to a device or function for capturing the game screen in real time and providing the image data to other modules or components.

[0007] "Analysis means" refers to a device or function used to process acquired image data and understand and analyze the situation within the game.

[0008] "Voice guide generation means" refers to a device or function that generates appropriate voice instructions for the player based on information obtained by the analysis means and outputs them as voice.

[0009] "Feedback generation means" refers to a device or function that generates haptic feedback corresponding to the game situation and provides the user with physical responses such as vibration or movement.

[0010] "Output means" refers to a device or function that transmits information using audio devices or haptic devices in order to provide audio guidance and feedback to the user.

[0011] "Voice input analysis means" refers to a device or function that acquires voice instructions from a user, analyzes them, and converts them into specific in-game operations.

[0012] "Operation control means" refers to a device or function that controls the game device in order to execute the in-game operation intended by the user, based on the results obtained by the voice input analysis means. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention provides a system that allows visually impaired individuals to enjoy games in the same way as sighted individuals. The server first captures the game screen in real time. The captured images are sent to an image processing module, where an analysis means identifies objects and situations within the game. Specifically, it analyzes important game elements in detail, such as the player's position, enemy movements, and the presence of obstacles.

[0035] Based on the analyzed information, the server creates voice instructions to communicate the situation to the user through a voice guide generation device. Additionally, a feedback generation device constructs haptic feedback appropriate to the game situation and transmits it to the user via the terminal.

[0036] The device receives voice commands and haptic feedback from the server and plays the voice commands using an audio output device such as a speaker. Simultaneously, it provides physical stimuli to the user through a haptic feedback device, allowing them to experience changes in the game.

[0037] Users play the game using voice instructions and haptic feedback from the server. When a user wants to perform a specific action, they send that instruction as voice input to their device. The device forwards this voice data to the server, which interprets the user's intent using voice input analysis tools.

[0038] Ultimately, the server uses control mechanisms to execute the user's desired actions within the game based on voice input. This allows users to perform complex game actions without relying on sight, enabling visually impaired individuals to enjoy the same gaming experience as sighted individuals.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The server captures the game screen in real time and generates image data. The captured data is sent to an image processing module.

[0042] Step 2:

[0043] The server uses an image processing module to analyze the position, movement, and state of in-game objects. This analysis identifies important elements such as the player, enemy characters, and obstacles, and helps to understand the game situation.

[0044] Step 3:

[0045] The server generates audio guidance based on the analyzed game situation. This audio guidance generation means constructs audio instructions to prompt the user to take appropriate action and organizes them as audio data.

[0046] Step 4:

[0047] The server uses a haptic feedback generation system to generate haptic feedback that corresponds to the situation in the game. This creates data to provide physical feedback to the user.

[0048] Step 5:

[0049] The server sends the generated audio data and haptic feedback data to the terminal.

[0050] Step 6:

[0051] The device receives audio data and uses an audio output device to play an audio guide for the user.

[0052] Step 7:

[0053] The device receives haptic feedback data and uses a corresponding feedback device to provide the user with physical feedback such as vibration or pressure.

[0054] Step 8:

[0055] Users control the game based on voice guidance and haptic feedback. They also speak voice commands into the microphone as needed.

[0056] Step 9:

[0057] The device receives the user's voice commands and sends that data to the server.

[0058] Step 10:

[0059] The server uses voice input analysis to analyze the user's voice commands and convert them into specific in-game actions.

[0060] Step 11:

[0061] The server controls the game device via an operation control means to execute user actions within the game based on voice commands.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] There is a problem in that visually impaired users have difficulty enjoying entertainment via electronic devices in the same way as sighted users. In particular, they face the challenge of not being able to fully enjoy interactive activities such as games because they cannot rely on visual information in situations that require complex operations or quick decision-making.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes data acquisition means for acquiring information from electronic devices in real time, data analysis means for analyzing the situation by analyzing the information, and guide generation means for generating voice instructions based on the analysis results. This enables visually impaired users to receive information using voice and touch and to operate the device accurately.

[0067] "Data acquisition means" refers to a function or device for collecting information from electronic devices in real time.

[0068] "Data analysis means" refers to a function or device that analyzes collected information and uses that information to analyze the situation.

[0069] "Guide generation means" refers to a function or device for creating voice instructions for the user based on the analysis results.

[0070] "Feedback generation means" refers to a function or device for creating tactile feedback based on analysis results.

[0071] "Signal output means" refers to a function or device that outputs voice instructions or haptic feedback to the user.

[0072] "Voice input analysis means" refers to a function or device that analyzes voice data acquired from a user and interprets its content.

[0073] "Operation control means" refers to a function or device for executing the operation intended by the user based on the analyzed voice data.

[0074] This system is designed to allow visually impaired users to enjoy interactive entertainment activities. The server first acquires information in real time from electronic devices such as games. This process utilizes standard data acquisition hardware, enabling the capture of high-resolution information.

[0075] The server transmits the acquired information to a data analysis module. This analysis uses software employing object recognition technology and other techniques to analyze various elements within the digital space. For example, it uses open-source image analysis libraries to identify the position and movement of objects.

[0076] Based on the analysis results, the server generates voice instructions using a voice guide generation system. In this process, external APIs and speech synthesis software are used to provide clear and specific instructions to the user. For example, the AI ​​model might be given the prompt, "Detect red and blue marks on the screen and extract the information."

[0077] In addition, haptic feedback is generated by feedback generation means, allowing the user to understand the situation through touch. This haptic feedback serves to attract the player's attention, for example, by using device vibrations.

[0078] The terminal conveys information to the user by playing voice instructions from the server through speakers or other means. It also uses haptic feedback devices to provide physical stimuli to the user, allowing them to experience real-time changes.

[0079] The user can issue instructions via voice input based on information from the server. The terminal acquires this voice and transmits it to the server. The server uses voice input analysis to interpret the voice data and performs operation control based on the analysis. In this way, the user can operate intuitively without relying on visual cues.

[0080] This system provides an environment in which visually impaired users can enjoy interactive activities, including complex operations, without relying on visual information.

[0081] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0082] Step 1:

[0083] The server acquires game screen information from electronic devices in real time. In this process, capture software is used to collect video data frame by frame. Game video data is acquired as input, and captured image frames are obtained as output. Specifically, the data is processed at 30 or 60 frames per second using a particular capture tool.

[0084] Step 2:

[0085] The server sends the acquired image frames to a data analysis module for analysis. This analysis process uses an image analysis library to detect objects and their placement. It receives captured images as input and generates object position information and motion information as output. Specifically, it identifies people and items based on the color patterns and shape features within the image.

[0086] Step 3:

[0087] The server generates audio guidance based on the analyzed information. It uses audio instruction generation software to convert text data into speech. It receives the analyzed text information as input and generates audio data as output. Specifically, it converts approaching enemies into specific warning voice messages such as "Enemies are approaching from the left."

[0088] Step 4:

[0089] The server configures haptic feedback based on the analysis results. This configuration uses an algorithm to set the vibration pattern of the haptic device. It utilizes the analysis results as input and generates control signals for haptic feedback as output. Specifically, it creates vibration patterns differentiated by the direction of the enemy and transmits feedback through the device.

[0090] Step 5:

[0091] The terminal plays voice commands sent from the server to the user using an audio output device. It receives audio data as input and plays the audio through a speaker as output. Specifically, it uses a high-quality speaker to deliver clear audio to the user.

[0092] Step 6:

[0093] The device provides physical stimulation to the user using a haptic device. It receives haptic feedback signals from the server as input and controls the device's vibration as output. For example, it controls the vibration motor of a gamepad to change the vibration according to the direction and distance of the enemy.

[0094] Step 7:

[0095] The user sends voice commands to the device using the voice input function. Voice recognition software converts this voice into text data and sends it to the device as input. The software receives the user's voice as input and generates text data as output. Specifically, it recognizes the voice command "move right" and generates the corresponding text.

[0096] Step 8:

[0097] The server analyzes voice text from the user and executes the intended action within the game. A voice input analysis module analyzes the voice data, and an action control module performs the corresponding action. It receives text data as input and executes in-game action as output. Specifically, it uses the keyboard input API to execute the action of "moving the game character to the right."

[0098] (Application Example 1)

[0099] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0100] A challenge exists in that visually impaired individuals have limited means of enjoying entertainment content without relying on visual information. Many games, movies, and other forms of entertainment rely on visual information, and as a result, visually impaired individuals cannot fully experience this content. Furthermore, there is a need for interactive systems that allow users to receive information in real time through sound and touch.

[0101] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0102] In this invention, the server includes data acquisition means for acquiring game information in real time, data analysis means for analyzing the data and extracting the game state, and audio signal generation means for generating audio signals based on the analysis results. This makes it possible for visually impaired people to experience entertainment content in real time through sound and touch.

[0103] A "data acquisition means" is a device that has the function of collecting information on entertainment content in real time.

[0104] A "data analysis device" is a device used to analyze acquired data and understand the state and status of the content.

[0105] A "voice signal generation means" is a device that generates voice signals based on data analysis results and has the function of transmitting appropriate information to the user.

[0106] A "tactile signal generation means" is a device that generates signals to provide physical feedback to the user based on analyzed information.

[0107] "Signal output means" refers to a device for outputting generated audio signals and tactile signals to the user.

[0108] A "voice signal analysis device" is a device that interprets voice signals obtained from a user and analyzes the user's intentions and requests.

[0109] "Operation control means" refers to a device that has the function of performing operations on entertainment content based on user requests.

[0110] An "entertainment robot" is a robot that provides entertainment content to visually impaired individuals and facilitates interactive experiences through sound and touch.

[0111] The system for implementing this invention begins with a server acquiring real-time game information and analyzing it using data analysis tools. The server captures the game screen in real time and extracts the necessary game information from it. This includes information such as character positions, enemy movements, and barriers. This is achieved using image processing software such as OpenCV.

[0112] Next, based on the analyzed information, the server generates an audio guide using an audio signal generation means. The generated audio guide is intended to inform the user of the game situation. The generated instructions are output as audio using speech synthesis software such as TextToSpeech.

[0113] Simultaneously, a haptic signal generation system generates haptic feedback based on the analysis results. This allows the user to progress through the game while feeling physical changes. The HapticFeedback system provides appropriate feedback to the user.

[0114] Subsequently, the user sends voice commands via a voice input analysis system, which the server interprets. Voice recognition software is used to clarify the user's intent and control in-game operations based on that intent.

[0115] This system enables visually impaired individuals to experience entertainment content in real time through sound and touch. Additionally, entertainment robots provide content to users, facilitating an interactive experience.

[0116] As a concrete example, this system can be used when visually impaired people enjoy real-time action games. The system notifies them of enemy appearances and obstacles in the game through voice and tactile feedback, allowing them to progress through the game by controlling their character according to the instructions.

[0117] Example prompt: "In this scene, the protagonist is searching for hidden treasure. Use the audio guide to indicate which direction to go. Also, use haptic feedback to show the location of obstacles."

[0118] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0119] Step 1:

[0120] The server uses cameras and capture devices to acquire the game screen as input images in order to collect real-time information about the game. Using this image data as input, the server analyzes the objects and situations in the game using image processing software (e.g., OpenCV), and extracts information about character positions, enemy movements, and obstacles. This analysis outputs data that describes the game situation in detail.

[0121] Step 2:

[0122] Using the analyzed game situation data as input, the server utilizes a voice signal generation means and a generation AI model (e.g., TextToSpeech) to generate an audio guide to inform the user of the game situation. This audio guide contains the necessary information to accurately convey the situation to the user and is output as an audio signal.

[0123] Step 3:

[0124] Simultaneously, the server uses haptic signal generation means to generate haptic feedback based on the analyzed information. This involves using a HapticFeedback system, which generates and outputs haptic signals such as vibrations so that the user can perceive physical changes within the game.

[0125] Step 4:

[0126] After audio and haptic signals are generated, the server sends these signals to the terminal, which then outputs them to the user via speakers or haptic devices using signal output means. Audio guidance is played through the audio output device, and haptic feedback is provided through devices worn by the user.

[0127] Step 5:

[0128] When a user enters a voice command, the terminal uses a microphone to capture the input voice. The server uses this voice data as input and analyzes it with voice recognition software (e.g., VoiceRecognition) using voice signal analysis tools to clarify the user's intent and output it.

[0129] Step 6:

[0130] After the server interprets the user's intent, it uses control mechanisms to execute the user's desired actions within the game. This allows the user to control the game in real time using voice commands, and the results of these actions are reflected in the game state.

[0131] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0132] This invention incorporates an emotion engine into a system that allows visually impaired individuals to enjoy games without relying on their vision. This engine recognizes the user's emotions in real time and adjusts the game experience accordingly. The server first captures the game screen in real time and transmits the image data to an analysis means via an image acquisition means. The analysis means analyzes the game situation in detail, including the character's position, enemy movements, and obstacle information.

[0133] Based on these analysis results, the server generates appropriate voice instructions using the voice guide generation means and creates haptic feedback using the feedback generation means. The generated voice instructions and haptic feedback are sent to the terminal, which then provides this information to the user. The voice guide is played back through an audio output device, and the haptic feedback is provided through a vibration device or the like.

[0134] Furthermore, the system incorporates a mechanism to acquire user voice input, analyze that voice, and implement specific game actions. An emotion engine is added to this, analyzing the user's emotional state in real time based on factors such as tone, speed, and language selection. Based on this emotional information, the server adjusts the content of voice commands and feedback to provide the user with the optimal gaming experience.

[0135] For example, if a user is dissatisfied or stressed by the game's difficulty, the emotion engine will detect this, and the server will adjust the game's difficulty appropriately. Conversely, if the system recognizes that the user is happy, it can increase positive feedback to maintain that positive emotion.

[0136] In this way, visually impaired individuals can enjoy games without relying on their sight by receiving appropriate instructions and feedback tailored to their emotional state.

[0137] The following describes the processing flow.

[0138] Step 1:

[0139] The server captures the game screen in real time and acquires data using an image acquisition device. The acquired image data is then sent to an analysis device.

[0140] Step 2:

[0141] The server uses analysis tools to analyze the game situation from captured image data. It identifies character positions, enemy movements, and obstacle placements to understand the in-game environment.

[0142] Step 3:

[0143] The server uses an audio guide generation mechanism to create an audio guide that communicates instructions based on the analyzed game situation to the user by structuring them as audio data.

[0144] Step 4:

[0145] The server uses a feedback generation mechanism to generate appropriate haptic feedback based on the analysis results. It creates feedback data such as vibration and pressure, preparing to communicate it directly to the user.

[0146] Step 5:

[0147] The server sends the generated audio guide and haptic feedback data to the device.

[0148] Step 6:

[0149] The device plays back the received audio guide to the user through an audio output device such as a speaker. Simultaneously, it provides physical feedback to the user through a haptic feedback device.

[0150] Step 7:

[0151] Users control the game based on voice guidance and haptic feedback. They can also use voice input to specify in-game actions when necessary.

[0152] Step 8:

[0153] The device receives the user's voice commands and sends that voice data to the server.

[0154] Step 9:

[0155] The server uses voice input analysis to analyze voice commands from the user and interpret them as specific game operations. Further action guidelines are then determined.

[0156] Step 10:

[0157] An emotion engine built into the server analyzes the user's tone, speed, and nonverbal cues to recognize the user's emotional state.

[0158] Step 11:

[0159] The server dynamically adjusts the content of voice guidance and feedback based on recognized emotions. It changes the difficulty level and instructions of in-game controls to provide a gaming experience that matches the user's emotions.

[0160] Step 12:

[0161] The server resends the adjusted data to the terminal, and the terminal reflects this in the user, continuing the feedback cycle.

[0162] (Example 2)

[0163] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0164] There is a need for systems that allow visually impaired people to enjoy games without relying on their vision, but current technology makes it difficult to provide an appropriate experience tailored to the user's emotional state. The challenge lies in effectively providing feedback that takes emotional states into account to create a more personalized gaming experience.

[0165] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0166] In this invention, the server includes an acquisition means for capturing information in real time, an analysis means for analyzing the information and analyzing the situation, and an adjustment means for analyzing the emotional state and adjusting the instructions and feedback content based on the emotional information. This allows for the provision of optimal instructions and feedback according to the user's emotional state, enabling them to enjoy the game without relying on their vision.

[0167] "Acquisition means" refers to a system element that provides the functionality to capture information in real time.

[0168] "Analysis means" refers to the elements of a system used to analyze captured information and interpret the situation.

[0169] "Instruction generation means" refers to a system element for generating instructions for the user based on the analysis results.

[0170] A "feedback generation means" is an element of a system for generating feedback based on analysis results.

[0171] "Output means" refers to elements of a system that provide instructions and feedback to the user.

[0172] An "input analysis means" is a system element that acquires input from the user and analyzes it.

[0173] A "control means" is an element of a system that controls the user's operation in response to acquired input.

[0174] "Adjustment mechanisms" refer to elements of a system that analyze the user's emotional state and adjust instructions and feedback accordingly.

[0175] This system utilizes a server equipped with an emotion engine to enable visually impaired individuals to enjoy games without relying on their vision. The server first acquires information in real time to understand the game's progress. Specifically, it uses a general-purpose image processing processor as hardware, and image processing libraries are used in the software.

[0176] The information acquired by the server is analyzed through analytical means to identify the location and status of characters in the game. The analysis utilizes generative AI models and machine learning libraries such as TENSORFLOW®.

[0177] Based on the analysis results, the server uses instruction generation and feedback generation means to create audio and haptic feedback. When generating audio instructions, speech synthesis software can be used, and services such as Google® Text-to-Speech can be utilized. The generated feedback is sent to the terminal via the WebSocket protocol and delivered to the user. The terminal provides this to the user, with audio instructions played back by an audio output device and haptic feedback presented through a vibration device.

[0178] The server also receives voice input from the user and analyzes it using input analysis tools. This process could utilize speech recognition technology and services like Amazon Transcribe.

[0179] Furthermore, the server analyzes the user's emotional state from their voice through adjustment mechanisms and optimizes the game experience. For example, if the user is feeling stressed, measures can be taken to lower the game's difficulty.

[0180] For example, if a user encounters a difficult situation during a game and says "This part is difficult" verbally, the server analyzes this information and uses the emotion engine to appropriately adjust the game's difficulty. An example of a prompt would be, "Create a program that generates voice instructions to adjust the game's difficulty and provides feedback to the user."

[0181] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0182] Step 1:

[0183] The server captures the game screen in real time. The input is the game's visual data, which is captured through a hardware processing unit. The server generates digital image data using an image processing library. The output is image data ready for analysis. This process provides fundamental data for understanding the game's progress.

[0184] Step 2:

[0185] The server analyzes the captured image data using an analysis tool. The input is the image data obtained in step 1, and the output is detailed analysis data including the positions of characters and enemies in the game, as well as information on obstacles. The server uses a generative AI model and machine learning libraries to precisely grasp the situation and perform specific actions to recognize the dynamic environment of the game.

[0186] Step 3:

[0187] The server generates voice instructions using an instruction generation mechanism based on the analysis results. The input is the analysis data generated in step 2, and the output is specific action instructions for the user, obtained as voice data. The server uses speech synthesis technology to compose the optimal instructions using a generation AI model, and converts them into speech using software such as Google Text-to-Speech. Through this process, the user can obtain instructions for their next action.

[0188] Step 4:

[0189] The server generates haptic feedback using a feedback generation mechanism based on the analysis results. The input is the analysis data obtained in step 2, and the output is a haptic signal that includes warnings for deviations, collisions, etc. The server uses a vibration device interface to generate specific vibration patterns, which are delivered to the user as haptic information. Through this process, the user gains a sensory understanding of the game environment.

[0190] Step 5:

[0191] The terminal provides the user with voice instructions and haptic feedback received from the server. Inputs include voice data and haptic signals obtained in steps 3 and 4, while outputs are feedback through a voice output device and a vibration device, respectively. The terminal performs specific actions, such as playing voice and generating vibrations. This allows the user to understand and respond to the situation in real time.

[0192] Step 6:

[0193] The server acquires the user's voice input and analyzes it using an input analysis tool. The input is voice data from the user, and the output is analysis data related to the user's intentions and emotional state. The server uses speech recognition software to convert the voice into text data, and then the emotion engine performs the specific actions to understand that emotional state.

[0194] Step 7:

[0195] The server optimizes instructions and feedback based on the user's emotional state. The input is the analysis data obtained in step 6, and the output is the adjusted instructions and feedback. The server uses a generative AI model to perform actions in real time to construct an emotionally responsive game experience. This allows the user to experience a game environment tailored to them.

[0196] (Application Example 2)

[0197] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0198] When visually impaired individuals navigate or engage in activities without relying on their sight, they face challenges in appropriately acquiring and understanding information about their surroundings. Furthermore, there is a lack of means to understand and respond to changes in their emotions during travel, providing appropriate guidance and support.

[0199] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0200] In this invention, the server includes information acquisition means, information analysis means, and instruction generation means. This allows visually impaired individuals to perceive their surroundings through sound and touch without relying on their vision, and to receive feedback and support in accordance with their emotions.

[0201] "Information acquisition means" refers to a device or system that collects information from the environment or target user in real time using visual, auditory, or other sensors.

[0202] An "information analysis tool" is a device or system that can analyze acquired information in detail and determine the environmental conditions, the location of objects, and the emotional state of the user.

[0203] "Instruction generation means" refers to a device or system for generating appropriate voice instructions or other feedback based on analysis results and providing them to the user.

[0204] A "feedback generation means" is a device or system that generates and provides tactile or other feedback tailored to the user's situation and emotions, based on the analysis results.

[0205] "Information output means" refers to a device or system for transmitting generated voice instructions and haptic feedback to the user.

[0206] "Input analysis means" refers to a device or system for acquiring voice input from a user and analyzing its intent and emotions.

[0207] "Control means" refers to a device or system for adjusting and controlling the user's operations and guidance based on the user's voice input or analyzed information.

[0208] An "emotion analysis device" is a device or system designed to analyze and determine a user's emotions in real time based on their voice and behavior.

[0209] "Adjustment means" refers to a device or system for appropriately adjusting the content of voice instructions and feedback according to the user's situation, based on the results of emotion analysis.

[0210] To implement this invention, the system uses the following elements: The server uses cameras and microphones as means of information acquisition to capture environmental information and user voice in real time. This collects the surrounding environment and user voice.

[0211] The server uses specialized analysis software to analyze the collected information. Specifically, it employs open-source computer vision libraries and speech analysis technologies. This allows for a detailed analysis of the environment and the user's emotional state.

[0212] Based on the analysis results, the server uses instruction generation means to generate appropriate voice instructions and feedback. Voice instructions are generated using text-to-speech conversion technology, and vibration feedback uses motor control technology. Instructions and feedback are provided to the user through information output means, which may include speakers or vibration devices.

[0213] Furthermore, voice input from the user is analyzed using input analysis means, and voice recognition software is introduced for this analysis. The results of this analysis are used to appropriately adjust user operation and guidance through control means. Control is performed using logic control within the program and external control units.

[0214] Furthermore, the server uses emotion analysis tools to determine the user's emotions in real time from their voice and behavior. This allows the server to instantly grasp the user's mental state and appropriately modify the generated feedback using adjustment tools.

[0215] For example, if a user feels anxious while walking through a park, the emotion analysis system will detect this, and the server will provide voice instructions in a gentler tone. These instructions might say something like, "Don't worry, this path has traffic lights and is safe." The prompt input to the generating AI model could be, "When the user is feeling anxious, generate a guidance message in a gentle tone to alleviate that emotion."

[0216] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0217] Step 1:

[0218] The server captures environmental information and user voice using information acquisition methods. It acquires video data captured by a camera and audio data collected by a microphone as input. This data allows for accurate understanding of the surrounding environment and the user's voice state.

[0219] Step 2:

[0220] The server analyzes the data acquired in the previous step using information analysis tools. It identifies the locations of people and obstacles from the input video data using an object detection algorithm, and extracts text information from the audio data using speech recognition technology. This allows for analysis of the environment and the user's speech.

[0221] Step 3:

[0222] The server uses emotion analysis tools to analyze the user's emotions in real time from voice data and behavior. The emotion analysis algorithm performs the analysis using voice tone, speed, keywords, etc. Based on the input voice data, it obtains output that reveals the user's mental state.

[0223] Step 4:

[0224] The server generates appropriate voice instructions using instruction generation means based on the information analyzed so far. Text-to-speech conversion software is used to create messages to be conveyed to the user. The generated voice instructions are designed to be helpful for movement and actions.

[0225] Step 5:

[0226] The server generates haptic feedback based on the analysis results using a feedback generation mechanism. It generates signals to control the vibration motor and transmits them to the user's device. This allows the user to physically perceive their surroundings without visual input.

[0227] Step 6:

[0228] The terminal provides the user with voice instructions and haptic feedback generated by an information output device. It receives voice files and vibration control signals from a server as input, and plays them back through a speaker and vibration device. The user then makes decisions or modifies their actions based on this feedback.

[0229] Step 7:

[0230] The user returns voice instructions and questions to the server via a voice input device. The system captures the user's spoken content as input and analyzes it using an input analysis tool. This allows the system to understand the user's recursive actions and prepare for the next step.

[0231] Step 8:

[0232] The server, using control mechanisms, generates prompt sentences to adjust situation-appropriate feedback based on the user's voice input and analysis of their emotional state, utilizing a generative AI model. This process creates data that provides optimal navigation and assistance for the user.

[0233] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0234] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0235] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0236] [Second Embodiment]

[0237] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0238] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0239] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0240] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0241] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0242] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0243] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0244] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0245] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0246] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0247] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0248] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0249] This invention provides a system that allows visually impaired individuals to enjoy games in the same way as sighted individuals. The server first captures the game screen in real time. The captured images are sent to an image processing module, where an analysis means identifies objects and situations within the game. Specifically, it analyzes important game elements in detail, such as the player's position, enemy movements, and the presence of obstacles.

[0250] Based on the analyzed information, the server creates voice instructions to communicate the situation to the user through a voice guide generation device. Additionally, a feedback generation device constructs haptic feedback appropriate to the game situation and transmits it to the user via the terminal.

[0251] The device receives voice commands and haptic feedback from the server and plays the voice commands using an audio output device such as a speaker. Simultaneously, it provides physical stimuli to the user through a haptic feedback device, allowing them to experience changes in the game.

[0252] Users play the game using voice instructions and haptic feedback from the server. When a user wants to perform a specific action, they send that instruction as voice input to their device. The device forwards this voice data to the server, which interprets the user's intent using voice input analysis tools.

[0253] Ultimately, the server uses control mechanisms to execute the user's desired actions within the game based on voice input. This allows users to perform complex game actions without relying on sight, enabling visually impaired individuals to enjoy the same gaming experience as sighted individuals.

[0254] The following describes the processing flow.

[0255] Step 1:

[0256] The server captures the game screen in real time and generates image data. The captured data is sent to an image processing module.

[0257] Step 2:

[0258] The server uses an image processing module to analyze the position, movement, and state of in-game objects. This analysis identifies important elements such as the player, enemy characters, and obstacles, and helps to understand the game situation.

[0259] Step 3:

[0260] The server generates audio guidance based on the analyzed game situation. This audio guidance generation means constructs audio instructions to prompt the user to take appropriate action and organizes them as audio data.

[0261] Step 4:

[0262] The server uses a haptic feedback generation system to generate haptic feedback that corresponds to the situation in the game. This creates data to provide physical feedback to the user.

[0263] Step 5:

[0264] The server sends the generated audio data and haptic feedback data to the terminal.

[0265] Step 6:

[0266] The device receives audio data and uses an audio output device to play an audio guide for the user.

[0267] Step 7:

[0268] The device receives haptic feedback data and uses a corresponding feedback device to provide the user with physical feedback such as vibration or pressure.

[0269] Step 8:

[0270] Users control the game based on voice guidance and haptic feedback. They also speak voice commands into the microphone as needed.

[0271] Step 9:

[0272] The device receives the user's voice commands and sends that data to the server.

[0273] Step 10:

[0274] The server uses voice input analysis to analyze the user's voice commands and convert them into specific in-game actions.

[0275] Step 11:

[0276] The server controls the game device via an operation control means to execute user actions within the game based on voice commands.

[0277] (Example 1)

[0278] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0279] There is a problem in that visually impaired users have difficulty enjoying entertainment via electronic devices in the same way as sighted users. In particular, they face the challenge of not being able to fully enjoy interactive activities such as games because they cannot rely on visual information in situations that require complex operations or quick decision-making.

[0280] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0281] In this invention, the server includes data acquisition means for acquiring information from electronic devices in real time, data analysis means for analyzing the situation by analyzing the information, and guide generation means for generating voice instructions based on the analysis results. This enables visually impaired users to receive information using voice and touch and to operate the device accurately.

[0282] "Data acquisition means" refers to a function or device for collecting information from electronic devices in real time.

[0283] "Data analysis means" refers to a function or device that analyzes collected information and uses that information to analyze the situation.

[0284] The "guide generation means" is a function or device for creating voice instructions for the user based on the analysis results.

[0285] The "feedback generation means" is a function or device for creating tactile feedback based on the analysis results.

[0286] The "signal output means" is a function or device for outputting voice instructions and tactile feedback to convey them to the user.

[0287] The "voice input analysis means" is a function or device for analyzing voice data acquired from the user and interpreting its content.

[0288] The "operation control means" is a function or device for executing the operation intended by the user based on the analyzed voice data.

[0289] This system is for visually impaired users to enjoy interactive entertainment activities. The server first acquires information from electronic devices such as games in real time. In this process, general data acquisition hardware can be used to capture high-definition information.

[0290] The server sends the acquired information to the data analysis module. For this analysis, software using object recognition technology and the like is used to analyze various elements in the virtual space. For example, an open-source image analysis library is used to identify the position and movement of objects.

[0291] Based on the analysis results, the server generates voice instructions using the voice guide generation means. At this time, external APIs and voice synthesis software are utilized for voice generation to provide clear and specific instructions to the user. For example, a prompt sentence such as "Detect red and blue marks from the screen and extract the information" is used for the AI model.

[0292] In addition, haptic feedback is generated by feedback generation means, allowing the user to understand the situation through touch. This haptic feedback serves to attract the player's attention, for example, by using device vibrations.

[0293] The terminal conveys information to the user by playing voice instructions from the server through speakers or other means. It also uses haptic feedback devices to provide physical stimuli to the user, allowing them to experience real-time changes.

[0294] The user can issue instructions via voice input based on information from the server. The terminal acquires this voice and transmits it to the server. The server uses voice input analysis to interpret the voice data and performs operation control based on the analysis. In this way, the user can operate intuitively without relying on visual cues.

[0295] This system provides an environment in which visually impaired users can enjoy interactive activities, including complex operations, without relying on visual information.

[0296] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0297] Step 1:

[0298] The server acquires game screen information from electronic devices in real time. In this process, capture software is used to collect video data frame by frame. Game video data is acquired as input, and captured image frames are obtained as output. Specifically, the data is processed at 30 or 60 frames per second using a particular capture tool.

[0299] Step 2:

[0300] The server sends the acquired image frames to a data analysis module for analysis. In this analysis process, an image analysis library is used to detect objects and placement information. It receives the captured image as input and generates object position information and moving object information as output. Specifically, it identifies people and items based on color patterns and shape features within the image.

[0301] Step 3:

[0302] The server generates an audio guide based on the analyzed information. It uses voice instruction generation software to convert text data into audio. It receives the text information of the analysis result as input and generates audio data as output. Specifically, it converts the approaching of an enemy into a specific warning audio such as "An enemy is coming from the left side."

[0303] Step 4:

[0304] The server constructs tactile feedback based on the analysis result. For this construction, an algorithm for setting the vibration pattern of a tactile device is used. It utilizes the analysis result as input and generates a control signal for tactile feedback as output. Specifically, it creates a vibration pattern differentiated by the direction of the enemy and transmits the feedback through the device.

[0305] Step 5:

[0306] The terminal plays the voice instruction sent from the server on a voice output device to notify the user. It receives audio data as input and plays the audio on a speaker as output. As a specific operation, it uses a high-quality speaker to deliver clear audio to the user.

[0307] Step 6:

[0308] The device provides physical stimulation to the user using a haptic device. It receives haptic feedback signals from the server as input and controls the device's vibration as output. For example, it controls the vibration motor of a gamepad to change the vibration according to the direction and distance of the enemy.

[0309] Step 7:

[0310] The user sends voice commands to the device using the voice input function. Voice recognition software converts this voice into text data and sends it to the device as input. The software receives the user's voice as input and generates text data as output. Specifically, it recognizes the voice command "move right" and generates the corresponding text.

[0311] Step 8:

[0312] The server analyzes voice text from the user and executes the intended action within the game. A voice input analysis module analyzes the voice data, and an action control module performs the corresponding action. It receives text data as input and executes in-game action as output. Specifically, it uses the keyboard input API to execute the action of "moving the game character to the right."

[0313] (Application Example 1)

[0314] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0315] A challenge exists in that visually impaired individuals have limited means of enjoying entertainment content without relying on visual information. Many games, movies, and other forms of entertainment rely on visual information, and as a result, visually impaired individuals cannot fully experience this content. Furthermore, there is a need for interactive systems that allow users to receive information in real time through sound and touch.

[0316] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0317] In this invention, the server includes data acquisition means for acquiring game information in real time, data analysis means for analyzing the data and extracting the game state, and audio signal generation means for generating audio signals based on the analysis results. This makes it possible for visually impaired people to experience entertainment content in real time through sound and touch.

[0318] A "data acquisition means" is a device that has the function of collecting information on entertainment content in real time.

[0319] A "data analysis device" is a device used to analyze acquired data and understand the state and status of the content.

[0320] A "voice signal generation means" is a device that generates voice signals based on data analysis results and has the function of transmitting appropriate information to the user.

[0321] A "tactile signal generation means" is a device that generates signals to provide physical feedback to the user based on analyzed information.

[0322] "Signal output means" refers to a device for outputting generated audio signals and tactile signals to the user.

[0323] A "voice signal analysis device" is a device that interprets voice signals obtained from a user and analyzes the user's intentions and requests.

[0324] "Operation control means" refers to a device that has the function of performing operations on entertainment content based on user requests.

[0325] An "entertainment robot" is a robot that provides entertainment content to visually impaired individuals and facilitates interactive experiences through sound and touch.

[0326] The system for implementing this invention begins with a server acquiring real-time game information and analyzing it using data analysis tools. The server captures the game screen in real time and extracts the necessary game information from it. This includes information such as character positions, enemy movements, and barriers. This is achieved using image processing software such as OpenCV.

[0327] Next, based on the analyzed information, the server generates an audio guide using an audio signal generation means. The generated audio guide is intended to inform the user of the game situation. The generated instructions are output as audio using speech synthesis software such as TextToSpeech.

[0328] Simultaneously, a haptic signal generation system generates haptic feedback based on the analysis results. This allows the user to progress through the game while feeling physical changes. The HapticFeedback system provides appropriate feedback to the user.

[0329] Subsequently, the user sends voice commands via a voice input analysis system, which the server interprets. Voice recognition software is used to clarify the user's intent and control in-game operations based on that intent.

[0330] This system enables visually impaired individuals to experience entertainment content in real time through sound and touch. Additionally, entertainment robots provide content to users, facilitating an interactive experience.

[0331] As a concrete example, this system can be used when visually impaired people enjoy real-time action games. The system notifies them of enemy appearances and obstacles in the game through voice and tactile feedback, allowing them to progress through the game by controlling their character according to the instructions.

[0332] Example prompt: "In this scene, the protagonist is searching for hidden treasure. Use the audio guide to indicate which direction to go. Also, use haptic feedback to show the location of obstacles."

[0333] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0334] Step 1:

[0335] The server uses cameras and capture devices to acquire the game screen as input images in order to collect real-time information about the game. Using this image data as input, the server analyzes the objects and situations in the game using image processing software (e.g., OpenCV), and extracts information about character positions, enemy movements, and obstacles. This analysis outputs data that describes the game situation in detail.

[0336] Step 2:

[0337] Using the analyzed game situation data as input, the server utilizes a voice signal generation means and a generation AI model (e.g., TextToSpeech) to generate an audio guide to inform the user of the game situation. This audio guide contains the necessary information to accurately convey the situation to the user and is output as an audio signal.

[0338] Step 3:

[0339] Simultaneously, the server uses haptic signal generation means to generate haptic feedback based on the analyzed information. This involves using a HapticFeedback system, which generates and outputs haptic signals such as vibrations so that the user can perceive physical changes within the game.

[0340] Step 4:

[0341] After audio and haptic signals are generated, the server sends these signals to the terminal, which then outputs them to the user via speakers or haptic devices using signal output means. Audio guidance is played through the audio output device, and haptic feedback is provided through devices worn by the user.

[0342] Step 5:

[0343] When a user enters a voice command, the terminal uses a microphone to capture the input voice. The server uses this voice data as input and analyzes it with voice recognition software (e.g., VoiceRecognition) using voice signal analysis tools to clarify the user's intent and output it.

[0344] Step 6:

[0345] After the server interprets the user's intent, it uses control mechanisms to execute the user's desired actions within the game. This allows the user to control the game in real time using voice commands, and the results of these actions are reflected in the game state.

[0346] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0347] This invention incorporates an emotion engine into a system that allows visually impaired individuals to enjoy games without relying on their vision. This engine recognizes the user's emotions in real time and adjusts the game experience accordingly. The server first captures the game screen in real time and transmits the image data to an analysis means via an image acquisition means. The analysis means analyzes the game situation in detail, including the character's position, enemy movements, and obstacle information.

[0348] Based on these analysis results, the server generates appropriate voice instructions using the voice guide generation means and creates haptic feedback using the feedback generation means. The generated voice instructions and haptic feedback are sent to the terminal, which then provides this information to the user. The voice guide is played back through an audio output device, and the haptic feedback is provided through a vibration device or the like.

[0349] Furthermore, the system incorporates a mechanism to acquire user voice input, analyze that voice, and implement specific game actions. An emotion engine is added to this, analyzing the user's emotional state in real time based on factors such as tone, speed, and language selection. Based on this emotional information, the server adjusts the content of voice commands and feedback to provide the user with the optimal gaming experience.

[0350] For example, if a user is dissatisfied or stressed by the game's difficulty, the emotion engine will detect this, and the server will adjust the game's difficulty appropriately. Conversely, if the system recognizes that the user is happy, it can increase positive feedback to maintain that positive emotion.

[0351] In this way, visually impaired individuals can enjoy games without relying on their sight by receiving appropriate instructions and feedback tailored to their emotional state.

[0352] The following describes the processing flow.

[0353] Step 1:

[0354] The server captures the game screen in real time and acquires data using an image acquisition device. The acquired image data is then sent to an analysis device.

[0355] Step 2:

[0356] The server uses analysis tools to analyze the game situation from captured image data. It identifies character positions, enemy movements, and obstacle placements to understand the in-game environment.

[0357] Step 3:

[0358] The server uses an audio guide generation mechanism to create an audio guide that communicates instructions based on the analyzed game situation to the user by structuring them as audio data.

[0359] Step 4:

[0360] The server uses a feedback generation mechanism to generate appropriate haptic feedback based on the analysis results. It creates feedback data such as vibration and pressure, preparing to communicate it directly to the user.

[0361] Step 5:

[0362] The server sends the generated audio guide and haptic feedback data to the device.

[0363] Step 6:

[0364] The device plays back the received audio guide to the user through an audio output device such as a speaker. Simultaneously, it provides physical feedback to the user through a haptic feedback device.

[0365] Step 7:

[0366] Users control the game based on voice guidance and haptic feedback. They can also use voice input to specify in-game actions when necessary.

[0367] Step 8:

[0368] The device receives the user's voice commands and sends that voice data to the server.

[0369] Step 9:

[0370] The server uses voice input analysis to analyze voice commands from the user and interpret them as specific game operations. Further action guidelines are then determined.

[0371] Step 10:

[0372] An emotion engine built into the server analyzes the user's tone, speed, and nonverbal cues to recognize the user's emotional state.

[0373] Step 11:

[0374] The server dynamically adjusts the content of voice guidance and feedback based on recognized emotions. It changes the difficulty level and instructions of in-game controls to provide a gaming experience that matches the user's emotions.

[0375] Step 12:

[0376] The server resends the adjusted data to the terminal, and the terminal reflects this in the user, continuing the feedback cycle.

[0377] (Example 2)

[0378] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0379] There is a need for systems that allow visually impaired people to enjoy games without relying on their vision, but current technology makes it difficult to provide an appropriate experience tailored to the user's emotional state. The challenge lies in effectively providing feedback that takes emotional states into account to create a more personalized gaming experience.

[0380] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0381] In this invention, the server includes an acquisition means for capturing information in real time, an analysis means for analyzing the information and analyzing the situation, and an adjustment means for analyzing the emotional state and adjusting the instructions and feedback content based on the emotional information. This allows for the provision of optimal instructions and feedback according to the user's emotional state, enabling them to enjoy the game without relying on their vision.

[0382] "Acquisition means" refers to a system element that provides the functionality to capture information in real time.

[0383] "Analysis means" refers to the elements of a system used to analyze captured information and interpret the situation.

[0384] "Instruction generation means" refers to a system element for generating instructions for the user based on the analysis results.

[0385] A "feedback generation means" is an element of a system for generating feedback based on analysis results.

[0386] "Output means" refers to elements of a system that provide instructions and feedback to the user.

[0387] An "input analysis means" is a system element that acquires input from the user and analyzes it.

[0388] A "control means" is an element of a system that controls the user's operation in response to acquired input.

[0389] "Adjustment mechanisms" refer to elements of a system that analyze the user's emotional state and adjust instructions and feedback accordingly.

[0390] This system utilizes a server equipped with an emotion engine to enable visually impaired individuals to enjoy games without relying on their vision. The server first acquires information in real time to understand the game's progress. Specifically, it uses a general-purpose image processing processor as hardware, and image processing libraries are used in the software.

[0391] The information acquired by the server is analyzed through analytical tools to identify the location and status of characters within the game. This analysis utilizes generative AI models and machine learning libraries such as TensorFlow.

[0392] Based on the analysis results, the server uses instruction generation and feedback generation means to create audio and haptic feedback. When generating audio instructions, speech synthesis software can be used, and services such as Google Text-to-Speech can be utilized. The generated feedback is sent to the terminal via the WebSocket protocol and reaches the user. The terminal provides this to the user, with audio instructions played back by an audio output device and haptic feedback presented through a vibration device.

[0393] The server also receives voice input from the user and analyzes it using input analysis tools. This process could utilize speech recognition technology and services like Amazon Transcribe.

[0394] Furthermore, the server analyzes the user's emotional state from their voice through adjustment mechanisms and optimizes the game experience. For example, if the user is feeling stressed, measures can be taken to lower the game's difficulty.

[0395] For example, if a user encounters a difficult situation during a game and says "This part is difficult" verbally, the server analyzes this information and uses the emotion engine to appropriately adjust the game's difficulty. An example of a prompt would be, "Create a program that generates voice instructions to adjust the game's difficulty and provides feedback to the user."

[0396] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0397] Step 1:

[0398] The server captures the game screen in real time. The input is the game's visual data, which is captured through a hardware processing unit. The server generates digital image data using an image processing library. The output is image data ready for analysis. This process provides fundamental data for understanding the game's progress.

[0399] Step 2:

[0400] The server analyzes the captured image data using an analysis tool. The input is the image data obtained in step 1, and the output is detailed analysis data including the positions of characters and enemies in the game, as well as information on obstacles. The server uses a generative AI model and machine learning libraries to precisely grasp the situation and perform specific actions to recognize the dynamic environment of the game.

[0401] Step 3:

[0402] The server generates voice instructions using an instruction generation mechanism based on the analysis results. The input is the analysis data generated in step 2, and the output is specific action instructions for the user, obtained as voice data. The server uses speech synthesis technology to compose the optimal instructions using a generation AI model, and converts them into speech using software such as Google Text-to-Speech. Through this process, the user can obtain instructions for their next action.

[0403] Step 4:

[0404] The server generates haptic feedback using a feedback generation mechanism based on the analysis results. The input is the analysis data obtained in step 2, and the output is a haptic signal that includes warnings for deviations, collisions, etc. The server uses a vibration device interface to generate specific vibration patterns, which are delivered to the user as haptic information. Through this process, the user gains a sensory understanding of the game environment.

[0405] Step 5:

[0406] The terminal provides the user with voice instructions and haptic feedback received from the server. Inputs include voice data and haptic signals obtained in steps 3 and 4, while outputs are feedback through a voice output device and a vibration device, respectively. The terminal performs specific actions, such as playing voice and generating vibrations. This allows the user to understand and respond to the situation in real time.

[0407] Step 6:

[0408] The server acquires the user's voice input and analyzes it using an input analysis tool. The input is voice data from the user, and the output is analysis data related to the user's intentions and emotional state. The server uses speech recognition software to convert the voice into text data, and then the emotion engine performs the specific actions to understand that emotional state.

[0409] Step 7:

[0410] The server optimizes instructions and feedback based on the user's emotional state. The input is the analysis data obtained in step 6, and the output is the adjusted instructions and feedback. The server uses a generative AI model to perform actions in real time to construct an emotionally responsive game experience. This allows the user to experience a game environment tailored to them.

[0411] (Application Example 2)

[0412] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0413] When visually impaired individuals navigate or engage in activities without relying on their sight, they face challenges in appropriately acquiring and understanding information about their surroundings. Furthermore, there is a lack of means to understand and respond to changes in their emotions during travel, providing appropriate guidance and support.

[0414] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0415] In this invention, the server includes information acquisition means, information analysis means, and instruction generation means. This allows visually impaired individuals to perceive their surroundings through sound and touch without relying on their vision, and to receive feedback and support in accordance with their emotions.

[0416] "Information acquisition means" refers to a device or system that collects information from the environment or target user in real time using visual, auditory, or other sensors.

[0417] An "information analysis tool" is a device or system that can analyze acquired information in detail and determine the environmental conditions, the location of objects, and the emotional state of the user.

[0418] "Instruction generation means" refers to a device or system for generating appropriate voice instructions or other feedback based on analysis results and providing them to the user.

[0419] A "feedback generation means" is a device or system that generates and provides tactile or other feedback tailored to the user's situation and emotions, based on the analysis results.

[0420] "Information output means" refers to a device or system for transmitting generated voice instructions and haptic feedback to the user.

[0421] "Input analysis means" refers to a device or system for acquiring voice input from a user and analyzing its intent and emotions.

[0422] "Control means" refers to a device or system for adjusting and controlling the user's operations and guidance based on the user's voice input or analyzed information.

[0423] An "emotion analysis device" is a device or system designed to analyze and determine a user's emotions in real time based on their voice and behavior.

[0424] "Adjustment means" refers to a device or system for appropriately adjusting the content of voice instructions and feedback according to the user's situation, based on the results of emotion analysis.

[0425] To implement this invention, the system uses the following elements: The server uses cameras and microphones as means of information acquisition to capture environmental information and user voice in real time. This collects the surrounding environment and user voice.

[0426] The server uses specialized analysis software to analyze the collected information. Specifically, it employs open-source computer vision libraries and speech analysis technologies. This allows for a detailed analysis of the environment and the user's emotional state.

[0427] Based on the analysis results, the server uses instruction generation means to generate appropriate voice instructions and feedback. Voice instructions are generated using text-to-speech conversion technology, and vibration feedback uses motor control technology. Instructions and feedback are provided to the user through information output means, which may include speakers or vibration devices.

[0428] Furthermore, voice input from the user is analyzed using input analysis means, and voice recognition software is introduced for this analysis. The results of this analysis are used to appropriately adjust user operation and guidance through control means. Control is performed using logic control within the program and external control units.

[0429] Furthermore, the server uses emotion analysis tools to determine the user's emotions in real time from their voice and behavior. This allows the server to instantly grasp the user's mental state and appropriately modify the generated feedback using adjustment tools.

[0430] For example, if a user feels anxious while walking through a park, the emotion analysis system will detect this, and the server will provide voice instructions in a gentler tone. These instructions might say something like, "Don't worry, this path has traffic lights and is safe." The prompt input to the generating AI model could be, "When the user is feeling anxious, generate a guidance message in a gentle tone to alleviate that emotion."

[0431] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0432] Step 1:

[0433] The server captures environmental information and user voice using information acquisition methods. It acquires video data captured by a camera and audio data collected by a microphone as input. This data allows for accurate understanding of the surrounding environment and the user's voice state.

[0434] Step 2:

[0435] The server analyzes the data acquired in the previous step using information analysis tools. It identifies the locations of people and obstacles from the input video data using an object detection algorithm, and extracts text information from the audio data using speech recognition technology. This allows for analysis of the environment and the user's speech.

[0436] Step 3:

[0437] The server uses emotion analysis tools to analyze the user's emotions in real time from voice data and behavior. The emotion analysis algorithm performs the analysis using voice tone, speed, keywords, etc. Based on the input voice data, it obtains output that reveals the user's mental state.

[0438] Step 4:

[0439] The server generates appropriate voice instructions using instruction generation means based on the information analyzed so far. Text-to-speech conversion software is used to create messages to be conveyed to the user. The generated voice instructions are designed to be helpful for movement and actions.

[0440] Step 5:

[0441] The server generates haptic feedback based on the analysis results using a feedback generation mechanism. It generates signals to control the vibration motor and transmits them to the user's device. This allows the user to physically perceive their surroundings without visual input.

[0442] Step 6:

[0443] The terminal provides the user with voice instructions and haptic feedback generated by an information output device. It receives voice files and vibration control signals from a server as input, and plays them back through a speaker and vibration device. The user then makes decisions or modifies their actions based on this feedback.

[0444] Step 7:

[0445] The user returns voice instructions and questions to the server via a voice input device. The system captures the user's spoken content as input and analyzes it using an input analysis tool. This allows the system to understand the user's recursive actions and prepare for the next step.

[0446] Step 8:

[0447] The server, using control mechanisms, generates prompt sentences to adjust situation-appropriate feedback based on the user's voice input and analysis of their emotional state, utilizing a generative AI model. This process creates data that provides optimal navigation and assistance for the user.

[0448] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0449] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0450] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0451] [Third Embodiment]

[0452] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0453] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0454] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0455] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0456] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0457] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0458] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0459] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0460] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0461] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0462] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0463] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0464] This invention provides a system that allows visually impaired individuals to enjoy games in the same way as sighted individuals. The server first captures the game screen in real time. The captured images are sent to an image processing module, where an analysis means identifies objects and situations within the game. Specifically, it analyzes important game elements in detail, such as the player's position, enemy movements, and the presence of obstacles.

[0465] Based on the analyzed information, the server creates voice instructions to communicate the situation to the user through a voice guide generation device. Additionally, a feedback generation device constructs haptic feedback appropriate to the game situation and transmits it to the user via the terminal.

[0466] The device receives voice commands and haptic feedback from the server and plays the voice commands using an audio output device such as a speaker. Simultaneously, it provides physical stimuli to the user through a haptic feedback device, allowing them to experience changes in the game.

[0467] Users play the game using voice instructions and haptic feedback from the server. When a user wants to perform a specific action, they send that instruction as voice input to their device. The device forwards this voice data to the server, which interprets the user's intent using voice input analysis tools.

[0468] Ultimately, the server uses control mechanisms to execute the user's desired actions within the game based on voice input. This allows users to perform complex game actions without relying on sight, enabling visually impaired individuals to enjoy the same gaming experience as sighted individuals.

[0469] The following describes the processing flow.

[0470] Step 1:

[0471] The server captures the game screen in real time and generates image data. The captured data is sent to an image processing module.

[0472] Step 2:

[0473] The server uses an image processing module to analyze the position, movement, and state of in-game objects. This analysis identifies important elements such as the player, enemy characters, and obstacles, and helps to understand the game situation.

[0474] Step 3:

[0475] The server generates audio guidance based on the analyzed game situation. This audio guidance generation means constructs audio instructions to prompt the user to take appropriate action and organizes them as audio data.

[0476] Step 4:

[0477] The server uses a haptic feedback generation system to generate haptic feedback that corresponds to the situation in the game. This creates data to provide physical feedback to the user.

[0478] Step 5:

[0479] The server sends the generated audio data and haptic feedback data to the terminal.

[0480] Step 6:

[0481] The device receives audio data and uses an audio output device to play an audio guide for the user.

[0482] Step 7:

[0483] The device receives haptic feedback data and uses a corresponding feedback device to provide the user with physical feedback such as vibration or pressure.

[0484] Step 8:

[0485] Users control the game based on voice guidance and haptic feedback. They also speak voice commands into the microphone as needed.

[0486] Step 9:

[0487] The device receives the user's voice commands and sends that data to the server.

[0488] Step 10:

[0489] The server uses voice input analysis to analyze the user's voice commands and convert them into specific in-game actions.

[0490] Step 11:

[0491] The server controls the game device via an operation control means to execute user actions within the game based on voice commands.

[0492] (Example 1)

[0493] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0494] There is a problem in that visually impaired users have difficulty enjoying entertainment via electronic devices in the same way as sighted users. In particular, they face the challenge of not being able to fully enjoy interactive activities such as games because they cannot rely on visual information in situations that require complex operations or quick decision-making.

[0495] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0496] In this invention, the server includes data acquisition means for acquiring information from electronic devices in real time, data analysis means for analyzing the situation by analyzing the information, and guide generation means for generating voice instructions based on the analysis results. This enables visually impaired users to receive information using voice and touch and to operate the device accurately.

[0497] "Data acquisition means" refers to a function or device for collecting information from electronic devices in real time.

[0498] "Data analysis means" refers to a function or device that analyzes collected information and uses that information to analyze the situation.

[0499] "Guide generation means" refers to a function or device for creating voice instructions for the user based on the analysis results.

[0500] "Feedback generation means" refers to a function or device for creating tactile feedback based on analysis results.

[0501] "Signal output means" refers to a function or device that outputs voice instructions or haptic feedback to the user.

[0502] "Voice input analysis means" refers to a function or device that analyzes voice data acquired from a user and interprets its content.

[0503] "Operation control means" refers to a function or device for executing the operation intended by the user based on the analyzed voice data.

[0504] This system is designed to allow visually impaired users to enjoy interactive entertainment activities. The server first acquires information in real time from electronic devices such as games. This process utilizes standard data acquisition hardware, enabling the capture of high-resolution information.

[0505] The server transmits the acquired information to a data analysis module. This analysis uses software employing object recognition technology and other techniques to analyze various elements within the digital space. For example, it uses open-source image analysis libraries to identify the position and movement of objects.

[0506] Based on the analysis results, the server generates voice instructions using a voice guide generation system. In this process, external APIs and speech synthesis software are used to provide clear and specific instructions to the user. For example, the AI ​​model might be given the prompt, "Detect red and blue marks on the screen and extract the information."

[0507] In addition, haptic feedback is generated by feedback generation means, allowing the user to understand the situation through touch. This haptic feedback serves to attract the player's attention, for example, by using device vibrations.

[0508] The terminal conveys information to the user by playing voice instructions from the server through speakers or other means. It also uses haptic feedback devices to provide physical stimuli to the user, allowing them to experience real-time changes.

[0509] The user can issue instructions via voice input based on information from the server. The terminal acquires this voice and transmits it to the server. The server uses voice input analysis to interpret the voice data and performs operation control based on the analysis. In this way, the user can operate intuitively without relying on visual cues.

[0510] This system provides an environment in which visually impaired users can enjoy interactive activities, including complex operations, without relying on visual information.

[0511] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0512] Step 1:

[0513] The server acquires game screen information from electronic devices in real time. In this process, capture software is used to collect video data frame by frame. Game video data is acquired as input, and captured image frames are obtained as output. Specifically, the data is processed at 30 or 60 frames per second using a particular capture tool.

[0514] Step 2:

[0515] The server sends the acquired image frames to a data analysis module for analysis. This analysis process uses an image analysis library to detect objects and their placement. It receives captured images as input and generates object position information and motion information as output. Specifically, it identifies people and items based on the color patterns and shape features within the image.

[0516] Step 3:

[0517] The server generates audio guidance based on the analyzed information. It uses audio instruction generation software to convert text data into speech. It receives the analyzed text information as input and generates audio data as output. Specifically, it converts approaching enemies into specific warning voice messages such as "Enemies are approaching from the left."

[0518] Step 4:

[0519] The server configures haptic feedback based on the analysis results. This configuration uses an algorithm to set the vibration pattern of the haptic device. It utilizes the analysis results as input and generates control signals for haptic feedback as output. Specifically, it creates vibration patterns differentiated by the direction of the enemy and transmits feedback through the device.

[0520] Step 5:

[0521] The terminal plays voice commands sent from the server to the user using an audio output device. It receives audio data as input and plays the audio through a speaker as output. Specifically, it uses a high-quality speaker to deliver clear audio to the user.

[0522] Step 6:

[0523] The device provides physical stimulation to the user using a haptic device. It receives haptic feedback signals from the server as input and controls the device's vibration as output. For example, it controls the vibration motor of a gamepad to change the vibration according to the direction and distance of the enemy.

[0524] Step 7:

[0525] The user sends voice commands to the device using the voice input function. Voice recognition software converts this voice into text data and sends it to the device as input. The software receives the user's voice as input and generates text data as output. Specifically, it recognizes the voice command "move right" and generates the corresponding text.

[0526] Step 8:

[0527] The server analyzes voice text from the user and executes the intended action within the game. A voice input analysis module analyzes the voice data, and an action control module performs the corresponding action. It receives text data as input and executes in-game action as output. Specifically, it uses the keyboard input API to execute the action of "moving the game character to the right."

[0528] (Application Example 1)

[0529] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0530] A challenge exists in that visually impaired individuals have limited means of enjoying entertainment content without relying on visual information. Many games, movies, and other forms of entertainment rely on visual information, and as a result, visually impaired individuals cannot fully experience this content. Furthermore, there is a need for interactive systems that allow users to receive information in real time through sound and touch.

[0531] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0532] In this invention, the server includes data acquisition means for acquiring game information in real time, data analysis means for analyzing the data and extracting the game state, and audio signal generation means for generating audio signals based on the analysis results. This makes it possible for visually impaired people to experience entertainment content in real time through sound and touch.

[0533] A "data acquisition means" is a device that has the function of collecting information on entertainment content in real time.

[0534] A "data analysis device" is a device used to analyze acquired data and understand the state and status of the content.

[0535] A "voice signal generation means" is a device that generates voice signals based on data analysis results and has the function of transmitting appropriate information to the user.

[0536] A "tactile signal generation means" is a device that generates signals to provide physical feedback to the user based on analyzed information.

[0537] "Signal output means" refers to a device for outputting generated audio signals and tactile signals to the user.

[0538] A "voice signal analysis device" is a device that interprets voice signals obtained from a user and analyzes the user's intentions and requests.

[0539] "Operation control means" refers to a device that has the function of performing operations on entertainment content based on user requests.

[0540] An "entertainment robot" is a robot that provides entertainment content to visually impaired individuals and facilitates interactive experiences through sound and touch.

[0541] The system for implementing this invention begins with a server acquiring real-time game information and analyzing it using data analysis tools. The server captures the game screen in real time and extracts the necessary game information from it. This includes information such as character positions, enemy movements, and barriers. This is achieved using image processing software such as OpenCV.

[0542] Next, based on the analyzed information, the server generates an audio guide using an audio signal generation means. The generated audio guide is intended to inform the user of the game situation. The generated instructions are output as audio using speech synthesis software such as TextToSpeech.

[0543] Simultaneously, a haptic signal generation system generates haptic feedback based on the analysis results. This allows the user to progress through the game while feeling physical changes. The HapticFeedback system provides appropriate feedback to the user.

[0544] Subsequently, the user sends voice commands via a voice input analysis system, which the server interprets. Voice recognition software is used to clarify the user's intent and control in-game operations based on that intent.

[0545] This system enables visually impaired individuals to experience entertainment content in real time through sound and touch. Additionally, entertainment robots provide content to users, facilitating an interactive experience.

[0546] As a concrete example, this system can be used when visually impaired people enjoy real-time action games. The system notifies them of enemy appearances and obstacles in the game through voice and tactile feedback, allowing them to progress through the game by controlling their character according to the instructions.

[0547] Example prompt: "In this scene, the protagonist is searching for hidden treasure. Use the audio guide to indicate which direction to go. Also, use haptic feedback to show the location of obstacles."

[0548] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0549] Step 1:

[0550] The server uses cameras and capture devices to acquire the game screen as input images in order to collect real-time information about the game. Using this image data as input, the server analyzes the objects and situations in the game using image processing software (e.g., OpenCV), and extracts information about character positions, enemy movements, and obstacles. This analysis outputs data that describes the game situation in detail.

[0551] Step 2:

[0552] Using the analyzed game situation data as input, the server utilizes a voice signal generation means and a generation AI model (e.g., TextToSpeech) to generate an audio guide to inform the user of the game situation. This audio guide contains the necessary information to accurately convey the situation to the user and is output as an audio signal.

[0553] Step 3:

[0554] Simultaneously, the server uses haptic signal generation means to generate haptic feedback based on the analyzed information. This involves using a HapticFeedback system, which generates and outputs haptic signals such as vibrations so that the user can perceive physical changes within the game.

[0555] Step 4:

[0556] After audio and haptic signals are generated, the server sends these signals to the terminal, which then outputs them to the user via speakers or haptic devices using signal output means. Audio guidance is played through the audio output device, and haptic feedback is provided through devices worn by the user.

[0557] Step 5:

[0558] When a user enters a voice command, the terminal uses a microphone to capture the input voice. The server uses this voice data as input and analyzes it with voice recognition software (e.g., VoiceRecognition) using voice signal analysis tools to clarify the user's intent and output it.

[0559] Step 6:

[0560] After the server interprets the user's intent, it uses control mechanisms to execute the user's desired actions within the game. This allows the user to control the game in real time using voice commands, and the results of these actions are reflected in the game state.

[0561] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0562] This invention incorporates an emotion engine into a system that allows visually impaired individuals to enjoy games without relying on their vision. This engine recognizes the user's emotions in real time and adjusts the game experience accordingly. The server first captures the game screen in real time and transmits the image data to an analysis means via an image acquisition means. The analysis means analyzes the game situation in detail, including the character's position, enemy movements, and obstacle information.

[0563] Based on these analysis results, the server generates appropriate voice instructions using the voice guide generation means and creates haptic feedback using the feedback generation means. The generated voice instructions and haptic feedback are sent to the terminal, which then provides this information to the user. The voice guide is played back through an audio output device, and the haptic feedback is provided through a vibration device or the like.

[0564] Furthermore, the system incorporates a mechanism to acquire user voice input, analyze that voice, and implement specific game actions. An emotion engine is added to this, analyzing the user's emotional state in real time based on factors such as tone, speed, and language selection. Based on this emotional information, the server adjusts the content of voice commands and feedback to provide the user with the optimal gaming experience.

[0565] For example, if a user is dissatisfied or stressed by the game's difficulty, the emotion engine will detect this, and the server will adjust the game's difficulty appropriately. Conversely, if the system recognizes that the user is happy, it can increase positive feedback to maintain that positive emotion.

[0566] In this way, visually impaired individuals can enjoy games without relying on their sight by receiving appropriate instructions and feedback tailored to their emotional state.

[0567] The following describes the processing flow.

[0568] Step 1:

[0569] The server captures the game screen in real time and acquires data using an image acquisition device. The acquired image data is then sent to an analysis device.

[0570] Step 2:

[0571] The server uses analysis tools to analyze the game situation from captured image data. It identifies character positions, enemy movements, and obstacle placements to understand the in-game environment.

[0572] Step 3:

[0573] The server uses an audio guide generation mechanism to create an audio guide that communicates instructions based on the analyzed game situation to the user by structuring them as audio data.

[0574] Step 4:

[0575] The server uses a feedback generation mechanism to generate appropriate haptic feedback based on the analysis results. It creates feedback data such as vibration and pressure, preparing to communicate it directly to the user.

[0576] Step 5:

[0577] The server sends the generated audio guide and haptic feedback data to the device.

[0578] Step 6:

[0579] The device plays back the received audio guide to the user through an audio output device such as a speaker. Simultaneously, it provides physical feedback to the user through a haptic feedback device.

[0580] Step 7:

[0581] Users control the game based on voice guidance and haptic feedback. They can also use voice input to specify in-game actions when necessary.

[0582] Step 8:

[0583] The device receives the user's voice commands and sends that voice data to the server.

[0584] Step 9:

[0585] The server uses voice input analysis to analyze voice commands from the user and interpret them as specific game operations. Further action guidelines are then determined.

[0586] Step 10:

[0587] An emotion engine built into the server analyzes the user's tone, speed, and nonverbal cues to recognize the user's emotional state.

[0588] Step 11:

[0589] The server dynamically adjusts the content of voice guidance and feedback based on recognized emotions. It changes the difficulty level and instructions of in-game controls to provide a gaming experience that matches the user's emotions.

[0590] Step 12:

[0591] The server resends the adjusted data to the terminal, and the terminal reflects this in the user, continuing the feedback cycle.

[0592] (Example 2)

[0593] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0594] There is a need for systems that allow visually impaired people to enjoy games without relying on their vision, but current technology makes it difficult to provide an appropriate experience tailored to the user's emotional state. The challenge lies in effectively providing feedback that takes emotional states into account to create a more personalized gaming experience.

[0595] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0596] In this invention, the server includes an acquisition means for capturing information in real time, an analysis means for analyzing the information and analyzing the situation, and an adjustment means for analyzing the emotional state and adjusting the instructions and feedback content based on the emotional information. This allows for the provision of optimal instructions and feedback according to the user's emotional state, enabling them to enjoy the game without relying on their vision.

[0597] "Acquisition means" refers to a system element that provides the functionality to capture information in real time.

[0598] "Analysis means" refers to the elements of a system used to analyze captured information and interpret the situation.

[0599] "Instruction generation means" refers to a system element for generating instructions for the user based on the analysis results.

[0600] A "feedback generation means" is an element of a system for generating feedback based on analysis results.

[0601] "Output means" refers to elements of a system that provide instructions and feedback to the user.

[0602] An "input analysis means" is a system element that acquires input from the user and analyzes it.

[0603] A "control means" is an element of a system that controls the user's operation in response to acquired input.

[0604] "Adjustment mechanisms" refer to elements of a system that analyze the user's emotional state and adjust instructions and feedback accordingly.

[0605] This system utilizes a server equipped with an emotion engine to enable visually impaired individuals to enjoy games without relying on their vision. The server first acquires information in real time to understand the game's progress. Specifically, it uses a general-purpose image processing processor as hardware, and image processing libraries are used in the software.

[0606] The information acquired by the server is analyzed through analytical tools to identify the location and status of characters within the game. This analysis utilizes generative AI models and machine learning libraries such as TensorFlow.

[0607] Based on the analysis results, the server uses instruction generation and feedback generation means to create audio and haptic feedback. When generating audio instructions, speech synthesis software can be used, and services such as Google Text-to-Speech can be utilized. The generated feedback is sent to the terminal via the WebSocket protocol and reaches the user. The terminal provides this to the user, with audio instructions played back by an audio output device and haptic feedback presented through a vibration device.

[0608] The server also receives voice input from the user and analyzes it using input analysis tools. This process could utilize speech recognition technology and services like Amazon Transcribe.

[0609] Furthermore, the server analyzes the user's emotional state from their voice through adjustment mechanisms and optimizes the game experience. For example, if the user is feeling stressed, measures can be taken to lower the game's difficulty.

[0610] For example, if a user encounters a difficult situation during a game and says "This part is difficult" verbally, the server analyzes this information and uses the emotion engine to appropriately adjust the game's difficulty. An example of a prompt would be, "Create a program that generates voice instructions to adjust the game's difficulty and provides feedback to the user."

[0611] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0612] Step 1:

[0613] The server captures the game screen in real time. The input is the game's visual data, which is captured through a hardware processing unit. The server generates digital image data using an image processing library. The output is image data ready for analysis. This process provides fundamental data for understanding the game's progress.

[0614] Step 2:

[0615] The server analyzes the captured image data using an analysis tool. The input is the image data obtained in step 1, and the output is detailed analysis data including the positions of characters and enemies in the game, as well as information on obstacles. The server uses a generative AI model and machine learning libraries to precisely grasp the situation and perform specific actions to recognize the dynamic environment of the game.

[0616] Step 3:

[0617] The server generates voice instructions using an instruction generation mechanism based on the analysis results. The input is the analysis data generated in step 2, and the output is specific action instructions for the user, obtained as voice data. The server uses speech synthesis technology to compose the optimal instructions using a generation AI model, and converts them into speech using software such as Google Text-to-Speech. Through this process, the user can obtain instructions for their next action.

[0618] Step 4:

[0619] The server generates haptic feedback using a feedback generation mechanism based on the analysis results. The input is the analysis data obtained in step 2, and the output is a haptic signal that includes warnings for deviations, collisions, etc. The server uses a vibration device interface to generate specific vibration patterns, which are delivered to the user as haptic information. Through this process, the user gains a sensory understanding of the game environment.

[0620] Step 5:

[0621] The terminal provides the user with voice instructions and haptic feedback received from the server. Inputs include voice data and haptic signals obtained in steps 3 and 4, while outputs are feedback through a voice output device and a vibration device, respectively. The terminal performs specific actions, such as playing voice and generating vibrations. This allows the user to understand and respond to the situation in real time.

[0622] Step 6:

[0623] The server acquires the user's voice input and analyzes it using an input analysis tool. The input is voice data from the user, and the output is analysis data related to the user's intentions and emotional state. The server uses speech recognition software to convert the voice into text data, and then the emotion engine performs the specific actions to understand that emotional state.

[0624] Step 7:

[0625] The server optimizes instructions and feedback based on the user's emotional state. The input is the analysis data obtained in step 6, and the output is the adjusted instructions and feedback. The server uses a generative AI model to perform actions in real time to construct an emotionally responsive game experience. This allows the user to experience a game environment tailored to them.

[0626] (Application Example 2)

[0627] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0628] When visually impaired individuals navigate or engage in activities without relying on their sight, they face challenges in appropriately acquiring and understanding information about their surroundings. Furthermore, there is a lack of means to understand and respond to changes in their emotions during travel, providing appropriate guidance and support.

[0629] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0630] In this invention, the server includes information acquisition means, information analysis means, and instruction generation means. This allows visually impaired individuals to perceive their surroundings through sound and touch without relying on their vision, and to receive feedback and support in accordance with their emotions.

[0631] "Information acquisition means" refers to a device or system that collects information from the environment or target user in real time using visual, auditory, or other sensors.

[0632] An "information analysis tool" is a device or system that can analyze acquired information in detail and determine the environmental conditions, the location of objects, and the emotional state of the user.

[0633] "Instruction generation means" refers to a device or system for generating appropriate voice instructions or other feedback based on analysis results and providing them to the user.

[0634] A "feedback generation means" is a device or system that generates and provides tactile or other feedback tailored to the user's situation and emotions, based on the analysis results.

[0635] "Information output means" refers to a device or system for transmitting generated voice instructions and haptic feedback to the user.

[0636] "Input analysis means" refers to a device or system for acquiring voice input from a user and analyzing its intent and emotions.

[0637] "Control means" refers to a device or system for adjusting and controlling the user's operations and guidance based on the user's voice input or analyzed information.

[0638] An "emotion analysis device" is a device or system designed to analyze and determine a user's emotions in real time based on their voice and behavior.

[0639] "Adjustment means" refers to a device or system for appropriately adjusting the content of voice instructions and feedback according to the user's situation, based on the results of emotion analysis.

[0640] To implement this invention, the system uses the following elements: The server uses cameras and microphones as means of information acquisition to capture environmental information and user voice in real time. This collects the surrounding environment and user voice.

[0641] The server uses specialized analysis software to analyze the collected information. Specifically, it employs open-source computer vision libraries and speech analysis technologies. This allows for a detailed analysis of the environment and the user's emotional state.

[0642] Based on the analysis results, the server uses instruction generation means to generate appropriate voice instructions and feedback. Voice instructions are generated using text-to-speech conversion technology, and vibration feedback uses motor control technology. Instructions and feedback are provided to the user through information output means, which may include speakers or vibration devices.

[0643] Furthermore, voice input from the user is analyzed using input analysis means, and voice recognition software is introduced for this analysis. The results of this analysis are used to appropriately adjust user operation and guidance through control means. Control is performed using logic control within the program and external control units.

[0644] Furthermore, the server uses emotion analysis tools to determine the user's emotions in real time from their voice and behavior. This allows the server to instantly grasp the user's mental state and appropriately modify the generated feedback using adjustment tools.

[0645] For example, if a user feels anxious while walking through a park, the emotion analysis system will detect this, and the server will provide voice instructions in a gentler tone. These instructions might say something like, "Don't worry, this path has traffic lights and is safe." The prompt input to the generating AI model could be, "When the user is feeling anxious, generate a guidance message in a gentle tone to alleviate that emotion."

[0646] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0647] Step 1:

[0648] The server captures environmental information and user voice using information acquisition methods. It acquires video data captured by a camera and audio data collected by a microphone as input. This data allows for accurate understanding of the surrounding environment and the user's voice state.

[0649] Step 2:

[0650] The server analyzes the data acquired in the previous step using information analysis tools. It identifies the locations of people and obstacles from the input video data using an object detection algorithm, and extracts text information from the audio data using speech recognition technology. This allows for analysis of the environment and the user's speech.

[0651] Step 3:

[0652] The server uses emotion analysis tools to analyze the user's emotions in real time from voice data and behavior. The emotion analysis algorithm performs the analysis using voice tone, speed, keywords, etc. Based on the input voice data, it obtains output that reveals the user's mental state.

[0653] Step 4:

[0654] The server generates appropriate voice instructions using instruction generation means based on the information analyzed so far. Text-to-speech conversion software is used to create messages to be conveyed to the user. The generated voice instructions are designed to be helpful for movement and actions.

[0655] Step 5:

[0656] The server generates haptic feedback based on the analysis results using a feedback generation mechanism. It generates signals to control the vibration motor and transmits them to the user's device. This allows the user to physically perceive their surroundings without visual input.

[0657] Step 6:

[0658] The terminal provides the user with voice instructions and haptic feedback generated by an information output device. It receives voice files and vibration control signals from a server as input, and plays them back through a speaker and vibration device. The user then makes decisions or modifies their actions based on this feedback.

[0659] Step 7:

[0660] The user returns voice instructions and questions to the server via a voice input device. The system captures the user's spoken content as input and analyzes it using an input analysis tool. This allows the system to understand the user's recursive actions and prepare for the next step.

[0661] Step 8:

[0662] The server, using control mechanisms, generates prompt sentences to adjust situation-appropriate feedback based on the user's voice input and analysis of their emotional state, utilizing a generative AI model. This process creates data that provides optimal navigation and assistance for the user.

[0663] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0664] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0665] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0666] [Fourth Embodiment]

[0667] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0668] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0669] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0670] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0671] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0672] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0673] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0674] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0675] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0676] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0677] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0678] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0679] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0680] This invention provides a system that allows visually impaired individuals to enjoy games in the same way as sighted individuals. The server first captures the game screen in real time. The captured images are sent to an image processing module, where an analysis means identifies objects and situations within the game. Specifically, it analyzes important game elements in detail, such as the player's position, enemy movements, and the presence of obstacles.

[0681] Based on the analyzed information, the server creates voice instructions to communicate the situation to the user through a voice guide generation device. Additionally, a feedback generation device constructs haptic feedback appropriate to the game situation and transmits it to the user via the terminal.

[0682] The device receives voice commands and haptic feedback from the server and plays the voice commands using an audio output device such as a speaker. Simultaneously, it provides physical stimuli to the user through a haptic feedback device, allowing them to experience changes in the game.

[0683] Users play the game using voice instructions and haptic feedback from the server. When a user wants to perform a specific action, they send that instruction as voice input to their device. The device forwards this voice data to the server, which interprets the user's intent using voice input analysis tools.

[0684] Ultimately, the server uses control mechanisms to execute the user's desired actions within the game based on voice input. This allows users to perform complex game actions without relying on sight, enabling visually impaired individuals to enjoy the same gaming experience as sighted individuals.

[0685] The following describes the processing flow.

[0686] Step 1:

[0687] The server captures the game screen in real time and generates image data. The captured data is sent to an image processing module.

[0688] Step 2:

[0689] The server uses an image processing module to analyze the position, movement, and state of in-game objects. This analysis identifies important elements such as the player, enemy characters, and obstacles, and helps to understand the game situation.

[0690] Step 3:

[0691] The server generates audio guidance based on the analyzed game situation. This audio guidance generation means constructs audio instructions to prompt the user to take appropriate action and organizes them as audio data.

[0692] Step 4:

[0693] The server uses a haptic feedback generation system to generate haptic feedback that corresponds to the situation in the game. This creates data to provide physical feedback to the user.

[0694] Step 5:

[0695] The server sends the generated audio data and haptic feedback data to the terminal.

[0696] Step 6:

[0697] The device receives audio data and uses an audio output device to play an audio guide for the user.

[0698] Step 7:

[0699] The device receives haptic feedback data and uses a corresponding feedback device to provide the user with physical feedback such as vibration or pressure.

[0700] Step 8:

[0701] Users control the game based on voice guidance and haptic feedback. They also speak voice commands into the microphone as needed.

[0702] Step 9:

[0703] The device receives the user's voice commands and sends that data to the server.

[0704] Step 10:

[0705] The server uses voice input analysis to analyze the user's voice commands and convert them into specific in-game actions.

[0706] Step 11:

[0707] The server controls the game device via an operation control means to execute user actions within the game based on voice commands.

[0708] (Example 1)

[0709] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0710] There is a problem in that visually impaired users have difficulty enjoying entertainment via electronic devices in the same way as sighted users. In particular, they face the challenge of not being able to fully enjoy interactive activities such as games because they cannot rely on visual information in situations that require complex operations or quick decision-making.

[0711] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0712] In this invention, the server includes data acquisition means for acquiring information from electronic devices in real time, data analysis means for analyzing the situation by analyzing the information, and guide generation means for generating voice instructions based on the analysis results. This enables visually impaired users to receive information using voice and touch and to operate the device accurately.

[0713] "Data acquisition means" refers to a function or device for collecting information from electronic devices in real time.

[0714] "Data analysis means" refers to a function or device that analyzes collected information and uses that information to analyze the situation.

[0715] "Guide generation means" refers to a function or device for creating voice instructions for the user based on the analysis results.

[0716] "Feedback generation means" refers to a function or device for creating tactile feedback based on analysis results.

[0717] "Signal output means" refers to a function or device that outputs voice instructions or haptic feedback to the user.

[0718] "Voice input analysis means" refers to a function or device that analyzes voice data acquired from a user and interprets its content.

[0719] "Operation control means" refers to a function or device for executing the operation intended by the user based on the analyzed voice data.

[0720] This system is designed to allow visually impaired users to enjoy interactive entertainment activities. The server first acquires information in real time from electronic devices such as games. This process utilizes standard data acquisition hardware, enabling the capture of high-resolution information.

[0721] The server transmits the acquired information to a data analysis module. This analysis uses software employing object recognition technology and other techniques to analyze various elements within the digital space. For example, it uses open-source image analysis libraries to identify the position and movement of objects.

[0722] Based on the analysis results, the server generates voice instructions using a voice guide generation system. In this process, external APIs and speech synthesis software are used to provide clear and specific instructions to the user. For example, the AI ​​model might be given the prompt, "Detect red and blue marks on the screen and extract the information."

[0723] In addition, haptic feedback is generated by feedback generation means, allowing the user to understand the situation through touch. This haptic feedback serves to attract the player's attention, for example, by using device vibrations.

[0724] The terminal conveys information to the user by playing voice instructions from the server through speakers or other means. It also uses haptic feedback devices to provide physical stimuli to the user, allowing them to experience real-time changes.

[0725] The user can issue instructions via voice input based on information from the server. The terminal acquires this voice and transmits it to the server. The server uses voice input analysis to interpret the voice data and performs operation control based on the analysis. In this way, the user can operate intuitively without relying on visual cues.

[0726] This system provides an environment in which visually impaired users can enjoy interactive activities, including complex operations, without relying on visual information.

[0727] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0728] Step 1:

[0729] The server acquires game screen information from electronic devices in real time. In this process, capture software is used to collect video data frame by frame. Game video data is acquired as input, and captured image frames are obtained as output. Specifically, the data is processed at 30 or 60 frames per second using a particular capture tool.

[0730] Step 2:

[0731] The server sends the acquired image frames to a data analysis module for analysis. This analysis process uses an image analysis library to detect objects and their placement. It receives captured images as input and generates object position information and motion information as output. Specifically, it identifies people and items based on the color patterns and shape features within the image.

[0732] Step 3:

[0733] The server generates audio guidance based on the analyzed information. It uses audio instruction generation software to convert text data into speech. It receives the analyzed text information as input and generates audio data as output. Specifically, it converts approaching enemies into specific warning voice messages such as "Enemies are approaching from the left."

[0734] Step 4:

[0735] The server configures haptic feedback based on the analysis results. This configuration uses an algorithm to set the vibration pattern of the haptic device. It utilizes the analysis results as input and generates control signals for haptic feedback as output. Specifically, it creates vibration patterns differentiated by the direction of the enemy and transmits feedback through the device.

[0736] Step 5:

[0737] The terminal plays voice commands sent from the server to the user using an audio output device. It receives audio data as input and plays the audio through a speaker as output. Specifically, it uses a high-quality speaker to deliver clear audio to the user.

[0738] Step 6:

[0739] The device provides physical stimulation to the user using a haptic device. It receives haptic feedback signals from the server as input and controls the device's vibration as output. For example, it controls the vibration motor of a gamepad to change the vibration according to the direction and distance of the enemy.

[0740] Step 7:

[0741] The user sends voice commands to the device using the voice input function. Voice recognition software converts this voice into text data and sends it to the device as input. The software receives the user's voice as input and generates text data as output. Specifically, it recognizes the voice command "move right" and generates the corresponding text.

[0742] Step 8:

[0743] The server analyzes voice text from the user and executes the intended action within the game. A voice input analysis module analyzes the voice data, and an action control module performs the corresponding action. It receives text data as input and executes in-game action as output. Specifically, it uses the keyboard input API to execute the action of "moving the game character to the right."

[0744] (Application Example 1)

[0745] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0746] A challenge exists in that visually impaired individuals have limited means of enjoying entertainment content without relying on visual information. Many games, movies, and other forms of entertainment rely on visual information, and as a result, visually impaired individuals cannot fully experience this content. Furthermore, there is a need for interactive systems that allow users to receive information in real time through sound and touch.

[0747] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0748] In this invention, the server includes data acquisition means for acquiring game information in real time, data analysis means for analyzing the data and extracting the game state, and audio signal generation means for generating audio signals based on the analysis results. This makes it possible for visually impaired people to experience entertainment content in real time through sound and touch.

[0749] A "data acquisition means" is a device that has the function of collecting information on entertainment content in real time.

[0750] A "data analysis device" is a device used to analyze acquired data and understand the state and status of the content.

[0751] A "voice signal generation means" is a device that generates voice signals based on data analysis results and has the function of transmitting appropriate information to the user.

[0752] A "tactile signal generation means" is a device that generates signals to provide physical feedback to the user based on analyzed information.

[0753] "Signal output means" refers to a device for outputting generated audio signals and tactile signals to the user.

[0754] A "voice signal analysis device" is a device that interprets voice signals obtained from a user and analyzes the user's intentions and requests.

[0755] "Operation control means" refers to a device that has the function of performing operations on entertainment content based on user requests.

[0756] An "entertainment robot" is a robot that provides entertainment content to visually impaired individuals and facilitates interactive experiences through sound and touch.

[0757] The system for implementing this invention begins with a server acquiring real-time game information and analyzing it using data analysis tools. The server captures the game screen in real time and extracts the necessary game information from it. This includes information such as character positions, enemy movements, and barriers. This is achieved using image processing software such as OpenCV.

[0758] Next, based on the analyzed information, the server generates an audio guide using an audio signal generation means. The generated audio guide is intended to inform the user of the game situation. The generated instructions are output as audio using speech synthesis software such as TextToSpeech.

[0759] Simultaneously, a haptic signal generation system generates haptic feedback based on the analysis results. This allows the user to progress through the game while feeling physical changes. The HapticFeedback system provides appropriate feedback to the user.

[0760] Subsequently, the user sends voice commands via a voice input analysis system, which the server interprets. Voice recognition software is used to clarify the user's intent and control in-game operations based on that intent.

[0761] This system enables visually impaired individuals to experience entertainment content in real time through sound and touch. Additionally, entertainment robots provide content to users, facilitating an interactive experience.

[0762] As a concrete example, this system can be used when visually impaired people enjoy real-time action games. The system notifies them of enemy appearances and obstacles in the game through voice and tactile feedback, allowing them to progress through the game by controlling their character according to the instructions.

[0763] Example prompt: "In this scene, the protagonist is searching for hidden treasure. Use the audio guide to indicate which direction to go. Also, use haptic feedback to show the location of obstacles."

[0764] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0765] Step 1:

[0766] The server uses cameras and capture devices to acquire the game screen as input images in order to collect real-time information about the game. Using this image data as input, the server analyzes the objects and situations in the game using image processing software (e.g., OpenCV), and extracts information about character positions, enemy movements, and obstacles. This analysis outputs data that describes the game situation in detail.

[0767] Step 2:

[0768] Using the analyzed game situation data as input, the server utilizes a voice signal generation means and a generation AI model (e.g., TextToSpeech) to generate an audio guide to inform the user of the game situation. This audio guide contains the necessary information to accurately convey the situation to the user and is output as an audio signal.

[0769] Step 3:

[0770] Simultaneously, the server uses haptic signal generation means to generate haptic feedback based on the analyzed information. This involves using a HapticFeedback system, which generates and outputs haptic signals such as vibrations so that the user can perceive physical changes within the game.

[0771] Step 4:

[0772] After audio and haptic signals are generated, the server sends these signals to the terminal, which then outputs them to the user via speakers or haptic devices using signal output means. Audio guidance is played through the audio output device, and haptic feedback is provided through devices worn by the user.

[0773] Step 5:

[0774] When a user enters a voice command, the terminal uses a microphone to capture the input voice. The server uses this voice data as input and analyzes it with voice recognition software (e.g., VoiceRecognition) using voice signal analysis tools to clarify the user's intent and output it.

[0775] Step 6:

[0776] After the server interprets the user's intent, it uses control mechanisms to execute the user's desired actions within the game. This allows the user to control the game in real time using voice commands, and the results of these actions are reflected in the game state.

[0777] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0778] This invention incorporates an emotion engine into a system that allows visually impaired individuals to enjoy games without relying on their vision. This engine recognizes the user's emotions in real time and adjusts the game experience accordingly. The server first captures the game screen in real time and transmits the image data to an analysis means via an image acquisition means. The analysis means analyzes the game situation in detail, including the character's position, enemy movements, and obstacle information.

[0779] Based on these analysis results, the server generates appropriate voice instructions using the voice guide generation means and creates haptic feedback using the feedback generation means. The generated voice instructions and haptic feedback are sent to the terminal, which then provides this information to the user. The voice guide is played back through an audio output device, and the haptic feedback is provided through a vibration device or the like.

[0780] Furthermore, the system incorporates a mechanism to acquire user voice input, analyze that voice, and implement specific game actions. An emotion engine is added to this, analyzing the user's emotional state in real time based on factors such as tone, speed, and language selection. Based on this emotional information, the server adjusts the content of voice commands and feedback to provide the user with the optimal gaming experience.

[0781] For example, if a user is dissatisfied or stressed by the game's difficulty, the emotion engine will detect this, and the server will adjust the game's difficulty appropriately. Conversely, if the system recognizes that the user is happy, it can increase positive feedback to maintain that positive emotion.

[0782] In this way, visually impaired individuals can enjoy games without relying on their sight by receiving appropriate instructions and feedback tailored to their emotional state.

[0783] The following describes the processing flow.

[0784] Step 1:

[0785] The server captures the game screen in real time and acquires data using an image acquisition device. The acquired image data is then sent to an analysis device.

[0786] Step 2:

[0787] The server uses analysis tools to analyze the game situation from captured image data. It identifies character positions, enemy movements, and obstacle placements to understand the in-game environment.

[0788] Step 3:

[0789] The server uses an audio guide generation mechanism to create an audio guide that communicates instructions based on the analyzed game situation to the user by structuring them as audio data.

[0790] Step 4:

[0791] The server uses a feedback generation mechanism to generate appropriate haptic feedback based on the analysis results. It creates feedback data such as vibration and pressure, preparing to communicate it directly to the user.

[0792] Step 5:

[0793] The server sends the generated audio guide and haptic feedback data to the device.

[0794] Step 6:

[0795] The device plays back the received audio guide to the user through an audio output device such as a speaker. Simultaneously, it provides physical feedback to the user through a haptic feedback device.

[0796] Step 7:

[0797] Users control the game based on voice guidance and haptic feedback. They can also use voice input to specify in-game actions when necessary.

[0798] Step 8:

[0799] The device receives the user's voice commands and sends that voice data to the server.

[0800] Step 9:

[0801] The server uses voice input analysis to analyze voice commands from the user and interpret them as specific game operations. Further action guidelines are then determined.

[0802] Step 10:

[0803] An emotion engine built into the server analyzes the user's tone, speed, and nonverbal cues to recognize the user's emotional state.

[0804] Step 11:

[0805] The server dynamically adjusts the content of voice guidance and feedback based on recognized emotions. It changes the difficulty level and instructions of in-game controls to provide a gaming experience that matches the user's emotions.

[0806] Step 12:

[0807] The server resends the adjusted data to the terminal, and the terminal reflects this in the user, continuing the feedback cycle.

[0808] (Example 2)

[0809] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0810] There is a need for systems that allow visually impaired people to enjoy games without relying on their vision, but current technology makes it difficult to provide an appropriate experience tailored to the user's emotional state. The challenge lies in effectively providing feedback that takes emotional states into account to create a more personalized gaming experience.

[0811] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0812] In this invention, the server includes an acquisition means for capturing information in real time, an analysis means for analyzing the information and analyzing the situation, and an adjustment means for analyzing the emotional state and adjusting the instructions and feedback content based on the emotional information. This allows for the provision of optimal instructions and feedback according to the user's emotional state, enabling them to enjoy the game without relying on their vision.

[0813] "Acquisition means" refers to a system element that provides the functionality to capture information in real time.

[0814] "Analysis means" refers to the elements of a system used to analyze captured information and interpret the situation.

[0815] "Instruction generation means" refers to a system element for generating instructions for the user based on the analysis results.

[0816] A "feedback generation means" is an element of a system for generating feedback based on analysis results.

[0817] "Output means" refers to elements of a system that provide instructions and feedback to the user.

[0818] An "input analysis means" is a system element that acquires input from the user and analyzes it.

[0819] A "control means" is an element of a system that controls the user's operation in response to acquired input.

[0820] "Adjustment mechanisms" refer to elements of a system that analyze the user's emotional state and adjust instructions and feedback accordingly.

[0821] This system utilizes a server equipped with an emotion engine to enable visually impaired individuals to enjoy games without relying on their vision. The server first acquires information in real time to understand the game's progress. Specifically, it uses a general-purpose image processing processor as hardware, and image processing libraries are used in the software.

[0822] The information acquired by the server is analyzed through analytical tools to identify the location and status of characters within the game. This analysis utilizes generative AI models and machine learning libraries such as TensorFlow.

[0823] Based on the analysis results, the server uses instruction generation and feedback generation means to create audio and haptic feedback. When generating audio instructions, speech synthesis software can be used, and services such as Google Text-to-Speech can be utilized. The generated feedback is sent to the terminal via the WebSocket protocol and reaches the user. The terminal provides this to the user, with audio instructions played back by an audio output device and haptic feedback presented through a vibration device.

[0824] The server also receives voice input from the user and analyzes it using input analysis tools. This process could utilize speech recognition technology and services like Amazon Transcribe.

[0825] Furthermore, the server analyzes the user's emotional state from their voice through adjustment mechanisms and optimizes the game experience. For example, if the user is feeling stressed, measures can be taken to lower the game's difficulty.

[0826] For example, if a user encounters a difficult situation during a game and says "This part is difficult" verbally, the server analyzes this information and uses the emotion engine to appropriately adjust the game's difficulty. An example of a prompt would be, "Create a program that generates voice instructions to adjust the game's difficulty and provides feedback to the user."

[0827] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0828] Step 1:

[0829] The server captures the game screen in real time. The input is the game's visual data, which is captured through a hardware processing unit. The server generates digital image data using an image processing library. The output is image data ready for analysis. This process provides fundamental data for understanding the game's progress.

[0830] Step 2:

[0831] The server analyzes the captured image data using an analysis tool. The input is the image data obtained in step 1, and the output is detailed analysis data including the positions of characters and enemies in the game, as well as information on obstacles. The server uses a generative AI model and machine learning libraries to precisely grasp the situation and perform specific actions to recognize the dynamic environment of the game.

[0832] Step 3:

[0833] The server generates voice instructions using an instruction generation mechanism based on the analysis results. The input is the analysis data generated in step 2, and the output is specific action instructions for the user, obtained as voice data. The server uses speech synthesis technology to compose the optimal instructions using a generation AI model, and converts them into speech using software such as Google Text-to-Speech. Through this process, the user can obtain instructions for their next action.

[0834] Step 4:

[0835] The server generates haptic feedback using a feedback generation mechanism based on the analysis results. The input is the analysis data obtained in step 2, and the output is a haptic signal that includes warnings for deviations, collisions, etc. The server uses a vibration device interface to generate specific vibration patterns, which are delivered to the user as haptic information. Through this process, the user gains a sensory understanding of the game environment.

[0836] Step 5:

[0837] The terminal provides the user with voice instructions and haptic feedback received from the server. Inputs include voice data and haptic signals obtained in steps 3 and 4, while outputs are feedback through a voice output device and a vibration device, respectively. The terminal performs specific actions, such as playing voice and generating vibrations. This allows the user to understand and respond to the situation in real time.

[0838] Step 6:

[0839] The server acquires the user's voice input and analyzes it using an input analysis tool. The input is voice data from the user, and the output is analysis data related to the user's intentions and emotional state. The server uses speech recognition software to convert the voice into text data, and then the emotion engine performs the specific actions to understand that emotional state.

[0840] Step 7:

[0841] The server optimizes instructions and feedback based on the user's emotional state. The input is the analysis data obtained in step 6, and the output is the adjusted instructions and feedback. The server uses a generative AI model to perform actions in real time to construct an emotionally responsive game experience. This allows the user to experience a game environment tailored to them.

[0842] (Application Example 2)

[0843] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0844] When visually impaired individuals navigate or engage in activities without relying on their sight, they face challenges in appropriately acquiring and understanding information about their surroundings. Furthermore, there is a lack of means to understand and respond to changes in their emotions during travel, providing appropriate guidance and support.

[0845] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0846] In this invention, the server includes information acquisition means, information analysis means, and instruction generation means. This allows visually impaired individuals to perceive their surroundings through sound and touch without relying on their vision, and to receive feedback and support in accordance with their emotions.

[0847] "Information acquisition means" refers to a device or system that collects information from the environment or target user in real time using visual, auditory, or other sensors.

[0848] An "information analysis tool" is a device or system that can analyze acquired information in detail and determine the environmental conditions, the location of objects, and the emotional state of the user.

[0849] "Instruction generation means" refers to a device or system for generating appropriate voice instructions or other feedback based on analysis results and providing them to the user.

[0850] A "feedback generation means" is a device or system that generates and provides tactile or other feedback tailored to the user's situation and emotions, based on the analysis results.

[0851] "Information output means" refers to a device or system for transmitting generated voice instructions and haptic feedback to the user.

[0852] "Input analysis means" refers to a device or system for acquiring voice input from a user and analyzing its intent and emotions.

[0853] "Control means" refers to a device or system for adjusting and controlling the user's operations and guidance based on the user's voice input or analyzed information.

[0854] An "emotion analysis device" is a device or system designed to analyze and determine a user's emotions in real time based on their voice and behavior.

[0855] "Adjustment means" refers to a device or system for appropriately adjusting the content of voice instructions and feedback according to the user's situation, based on the results of emotion analysis.

[0856] To implement this invention, the system uses the following elements: The server uses cameras and microphones as means of information acquisition to capture environmental information and user voice in real time. This collects the surrounding environment and user voice.

[0857] The server uses specialized analysis software to analyze the collected information. Specifically, it employs open-source computer vision libraries and speech analysis technologies. This allows for a detailed analysis of the environment and the user's emotional state.

[0858] Based on the analysis results, the server uses instruction generation means to generate appropriate voice instructions and feedback. Voice instructions are generated using text-to-speech conversion technology, and vibration feedback uses motor control technology. Instructions and feedback are provided to the user through information output means, which may include speakers or vibration devices.

[0859] Furthermore, voice input from the user is analyzed using input analysis means, and voice recognition software is introduced for this analysis. The results of this analysis are used to appropriately adjust user operation and guidance through control means. Control is performed using logic control within the program and external control units.

[0860] Furthermore, the server uses emotion analysis tools to determine the user's emotions in real time from their voice and behavior. This allows the server to instantly grasp the user's mental state and appropriately modify the generated feedback using adjustment tools.

[0861] For example, if a user feels anxious while walking through a park, the emotion analysis system will detect this, and the server will provide voice instructions in a gentler tone. These instructions might say something like, "Don't worry, this path has traffic lights and is safe." The prompt input to the generating AI model could be, "When the user is feeling anxious, generate a guidance message in a gentle tone to alleviate that emotion."

[0862] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0863] Step 1:

[0864] The server captures environmental information and user voice using information acquisition methods. It acquires video data captured by a camera and audio data collected by a microphone as input. This data allows for accurate understanding of the surrounding environment and the user's voice state.

[0865] Step 2:

[0866] The server analyzes the data acquired in the previous step using information analysis tools. It identifies the locations of people and obstacles from the input video data using an object detection algorithm, and extracts text information from the audio data using speech recognition technology. This allows for analysis of the environment and the user's speech.

[0867] Step 3:

[0868] The server uses emotion analysis tools to analyze the user's emotions in real time from voice data and behavior. The emotion analysis algorithm performs the analysis using voice tone, speed, keywords, etc. Based on the input voice data, it obtains output that reveals the user's mental state.

[0869] Step 4:

[0870] The server generates appropriate voice instructions using instruction generation means based on the information analyzed so far. Text-to-speech conversion software is used to create messages to be conveyed to the user. The generated voice instructions are designed to be helpful for movement and actions.

[0871] Step 5:

[0872] The server generates haptic feedback based on the analysis results using a feedback generation mechanism. It generates signals to control the vibration motor and transmits them to the user's device. This allows the user to physically perceive their surroundings without visual input.

[0873] Step 6:

[0874] The terminal provides the user with voice instructions and haptic feedback generated by an information output device. It receives voice files and vibration control signals from a server as input, and plays them back through a speaker and vibration device. The user then makes decisions or modifies their actions based on this feedback.

[0875] Step 7:

[0876] The user returns voice instructions and questions to the server via a voice input device. The system captures the user's spoken content as input and analyzes it using an input analysis tool. This allows the system to understand the user's recursive actions and prepare for the next step.

[0877] Step 8:

[0878] The server, using control mechanisms, generates prompt sentences to adjust situation-appropriate feedback based on the user's voice input and analysis of their emotional state, utilizing a generative AI model. This process creates data that provides optimal navigation and assistance for the user.

[0879] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0880] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0881] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0882] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0883] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0884] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0885] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0886] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0887] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0888] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0889] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0890] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0891] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0892] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0893] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0894] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0895] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0896] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0897] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0898] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0899] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0900] The following is further disclosed regarding the embodiments described above.

[0901] (Claim 1)

[0902] A means of acquiring images to capture game information in real time,

[0903] An analysis means for analyzing the game situation by analyzing the aforementioned image,

[0904] A voice guide generation means that generates voice instructions based on the analysis results,

[0905] A feedback generation means that generates tactile feedback based on the analysis results,

[0906] Output means for outputting the aforementioned voice instructions and haptic feedback,

[0907] A voice input analysis means that acquires and analyzes the user's voice input,

[0908] Operation control means that controls the user's game operations in response to the aforementioned voice input,

[0909] A system that includes this.

[0910] (Claim 2)

[0911] The system according to claim 1, wherein the analysis means analyzes game information including character positions, enemy positions, and obstacle information.

[0912] (Claim 3)

[0913] The system according to claim 1, wherein the voice guide generation means generates specific instructions according to the user's situation and converts that information into voice.

[0914] "Example 1"

[0915] (Claim 1)

[0916] A data acquisition method that obtains information from electronic devices in real time,

[0917] A data analysis means for analyzing the situation by analyzing the aforementioned information,

[0918] A guide generation means that generates voice instructions based on the analysis results,

[0919] A feedback generation means that generates tactile feedback based on the analysis results,

[0920] The signal output means for outputting the aforementioned voice instructions and haptic feedback,

[0921] A voice input analysis means that acquires and analyzes the user's voice data,

[0922] An operation control means that controls user operations in response to the aforementioned voice data,

[0923] A system that includes this.

[0924] (Claim 2)

[0925] The system according to claim 1, wherein the data analysis means analyzes information including object position, moving body position, and fault information.

[0926] (Claim 3)

[0927] The system according to claim 1, wherein the guide generation means generates specific instructions according to the user's state and converts that information into speech.

[0928] "Application Example 1"

[0929] (Claim 1)

[0930] A data acquisition method for obtaining game information in real time,

[0931] A data analysis means for analyzing the aforementioned data and extracting the game state,

[0932] Audio signal generation means that generates an audio signal based on the analysis results,

[0933] A tactile signal generation means that generates tactile signals based on the analysis results,

[0934] The signal output means for outputting the aforementioned audio signal and tactile signal,

[0935] A voice signal analysis means for acquiring and analyzing the user's voice signal,

[0936] Operation control means that controls user operations in response to the aforementioned audio signal,

[0937] Including robots that provide entertainment content for the visually impaired,

[0938] A system that includes this.

[0939] (Claim 2)

[0940] The analysis means analyzes game information including the character's position, the enemy's position, and information about barriers.

[0941] Generates audio and haptic guidance.

[0942] The system according to claim 1.

[0943] (Claim 3)

[0944] The aforementioned audio signal generation means concretizes instructions according to the user's context and converts those instructions into audio output.

[0945] Simultaneously, it works in conjunction with tactile signal generation means to provide real-time feedback through an entertainment robot.

[0946] The system according to claim 1.

[0947] "Example 2 of combining an emotion engine"

[0948] (Claim 1)

[0949] A means of capturing information in real time,

[0950] An analytical means for analyzing the situation by analyzing the aforementioned information,

[0951] Instruction generation means that generates instructions based on the analysis results,

[0952] A feedback generation means that generates feedback based on the analysis results,

[0953] Output means for outputting the aforementioned instructions and feedback,

[0954] An input analysis means that acquires and analyzes user input,

[0955] Control means for controlling the user's operation in response to the aforementioned input,

[0956] An adjustment means for analyzing emotional states and adjusting instructions and feedback based on emotional information,

[0957] A system that includes this.

[0958] (Claim 2)

[0959] The system according to claim 1, wherein the analysis means analyzes information including location information, movement information, and fault information.

[0960] (Claim 3)

[0961] The system according to claim 1, wherein the instruction generation means generates specific instructions according to the user's situation and converts that information into speech.

[0962] "Application example 2 when combining with an emotional engine"

[0963] (Claim 1)

[0964] A means of acquiring information to capture game information in real time,

[0965] Information analysis means for analyzing the situation by analyzing the aforementioned information,

[0966] Instruction generation means that generates voice instructions based on the analysis results,

[0967] A feedback generation means that generates tactile feedback based on the analysis results,

[0968] Information output means for outputting the aforementioned voice instructions and haptic feedback,

[0969] An input analysis means that acquires and analyzes the user's voice input,

[0970] Control means for controlling user operations in response to the aforementioned voice input,

[0971] Equipped with emotion analysis capabilities to analyze user emotions in real time,

[0972] Based on the emotion analysis results, an adjustment means for adjusting voice instructions and haptic feedback,

[0973] A system that includes this.

[0974] (Claim 2)

[0975] The system according to claim 1, wherein the information analysis means analyzes information including object position and obstacle information, and generates situation-appropriate feedback based on the user's emotional state.

[0976] (Claim 3)

[0977] The system according to claim 1, wherein the adjustment means generates specific instructions based on the user's emotions, converts the information into voice, and provides it as feedback. [Explanation of symbols]

[0978] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring images to capture game information in real time, An analysis means for analyzing the game situation by analyzing the aforementioned image, A voice guide generation means that generates voice instructions based on the analysis results, A feedback generation means that generates tactile feedback based on the analysis results, Output means for outputting the aforementioned voice instructions and haptic feedback, A voice input analysis means that acquires and analyzes the user's voice input, Operation control means that controls the user's game operations in response to the aforementioned voice input, A system that includes this.

2. The system according to claim 1, wherein the analysis means analyzes game information including character positions, enemy positions, and obstacle information.

3. The system according to claim 1, wherein the voice guide generation means generates specific instructions according to the user's situation and converts that information into voice.

Images