system

Abstract

We provide the system. [Solution] Means for generating a virtual reality environment, A means of tracking user actions and reflecting them in a virtual reality space, An artificial intelligence method that recognizes and evaluates the user's voice in real time, Means for generating visual or auditory feedback based on the aforementioned evaluation, A means of sharing user-generated content and receiving feedback from other users, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional karaoke systems and music expression platforms, it is difficult for users to obtain an immersive virtual concert experience, and it is impossible to receive specific feedback in real time to improve singing skills. There is also a problem that there is a lack of means to widely share performances and increase opportunities for self-expression through evaluations from others.

Means for Solving the Problems

[0005] This invention provides a system that generates a virtual reality environment and tracks user actions, reflecting them in the virtual reality space. Furthermore, it includes means for generating visual or auditory feedback by using artificial intelligence to recognize and evaluate the user's voice in real time, based on pitch, rhythm, and volume. The invention also solves these problems by providing means for users to share their generated content with others and receive feedback from other users.

[0006] A "virtual reality environment" is a three-dimensional computer-generated environment that, through a dedicated device, creates a visual and experiential illusion of being real.

[0007] "Tracking" is a technology that detects a user's movements and location, processes this data in real time as digital data, and reflects it in a corresponding virtual space.

[0008] Artificial intelligence is a computer system that uses machine learning and data analysis technologies to evaluate user performance and make decisions based on specific criteria.

[0009] "Feedback" refers to the responses or advice a system provides to a user's actions or input, and can take the form of visual or auditory feedback.

[0010] "Sharing" refers to the act of making user-generated content publicly available on a platform accessible to other users, and receiving ratings and comments. [Brief explanation of the drawing]

[0011] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0012] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0013] First, let's explain the terminology used in the following explanation.

[0014] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0015] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0016] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs, various parameters, and the like. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0017] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor, an antenna, and the like. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0019] [First Embodiment]

[0020] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0021] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0024] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0027] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0031] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0032] The system of the present invention is configured to enable users to experience an immersive concert in a virtual reality space. This system functions through the cooperation of a server, a terminal, and the user. The following describes each element and specific embodiments.

[0033] The server is responsible for generating the virtual reality environment and constructing themes and stage designs according to the user's selections. When a user chooses a specific stage design, the server prepares 3D models and effect data corresponding to that design and streams them to the terminal. At this time, the server receives the user's voice data and performs real-time voice analysis using artificial intelligence to evaluate the singing.

[0034] The terminal allows users to access the system using a virtual reality device and interact with it in a natural way. The terminal has the capability to capture user movements and reflect them in the virtual space. It also receives feedback based on user performance evaluations sent from the server and presents it to the user in an appropriate format. For example, if a user sings off-key, that part is visually highlighted.

[0035] After logging into the system, users can realize their performance in a virtual reality space while making various settings. Users can select songs from the content library provided by the system and start singing. They can also use their devices to receive real-time feedback, adjust their singing, and strive to improve their skills.

[0036] As a concrete example, suppose the user selects a rock concert stage. In this case, the server generates a rock-themed stage set, and the device reflects the user's movements in real time. When the user sings a song, artificial intelligence analyzes the voice in real time and provides accurate feedback. Based on this feedback, the user improves their singing.

[0037] Furthermore, this system also includes a feature that allows users to share the performance they generate with other users. The server stores user performance data and makes it available for sharing within the community. This expands opportunities for users to express themselves through feedback and evaluations from other users.

[0038] The following describes the processing flow.

[0039] Step 1:

[0040] The user logs into the system. The terminal obtains the user's authentication information from the input screen and sends it to the server. The server uses the received authentication information to refer to the database and authenticate the user's account.

[0041] Step 2:

[0042] The user selects a theme and stage for the virtual concert. Based on the user's selection, the server prepares the corresponding 3D models and effect data and gets ready to transfer them to the terminal.

[0043] Step 3:

[0044] The device initializes the VR device and tracks the user's position and movements in real time. This ensures that the user's movements are accurately reflected in the virtual reality space.

[0045] Step 4:

[0046] The user selects a song to sing. The device requests the selected song data from the server and stores the data in the cache as needed.

[0047] Step 5:

[0048] When the user starts singing, the device captures the user's voice from the microphone and streams that audio data to the server in real time.

[0049] Step 6:

[0050] The server uses artificial intelligence to analyze the received audio data and evaluates the user's singing based on various evaluation criteria such as pitch, rhythm, and volume.

[0051] Step 7:

[0052] The server generates evaluation results and sends them to the terminal as feedback. The terminal then presents this feedback to the user visually or audibly.

[0053] Step 8:

[0054] The user adjusts their singing based on the feedback and practices further. If necessary, the device records the user's performance and saves it for later playback.

[0055] Step 9:

[0056] If a user chooses to share their performance with the community, the server stores that data and makes it accessible to other users.

[0057] Step 10:

[0058] Other users can view the shared performance and leave comments and ratings. The server notifies the user who originally posted the performance of any newly added comments or ratings.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] Current virtual reality technologies make it difficult for users to experience immersive, real-time performances. Furthermore, there is a lack of immediate feedback to effectively promote individual user skill improvement. Additionally, there are limited means for sharing generated performances with others and stimulating communication.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes means having the function of generating a virtual reality space, means having the function of recording the user's activities and reflecting them in the virtual environment space, and an artificial intelligence device that recognizes the user's voice and performs analysis and evaluation in real time. As a result, the user can enjoy an immersive performance experience in real time, improve their skills by receiving immediate feedback, and effectively share the generated performance with other users.

[0064] A "virtual reality space" is a three-dimensional environment generated by a computer, in which users can have an immersive experience in real time.

[0065] An "information processing device" is an electronic device that performs data calculations, analysis, streaming, and other processing to enable virtual reality experiences.

[0066] "User activity" refers to the collective actions and voice inputs performed by the user within the virtual reality space, which are reflected in the virtual environment through the interface.

[0067] An "artificial intelligence device" is a set of software or hardware that analyzes user input such as voice and movement in real time through data analysis and learning, and provides evaluation and feedback.

[0068] "Subject" refers to a theme or concept in a virtual reality space, and is an element that expresses different styles and atmospheres that can be selected by the user.

[0069] "Spatial design" refers to the arrangement and design of environments and stages within a virtual reality space, and is an element that should be customized according to the user's preferences.

[0070] This invention is a system that enables users to conduct immersive performance experiences in a virtual reality space. Specific embodiments of each element are shown below.

[0071] The server plays a central role in generating the virtual reality space. Based on the theme selected by the user, the server prepares 3D designs and effects. For example, if the user chooses a rock concert theme, the server generates data for a rock-style stage set and streams it to the terminal. The server uses a generative AI model, particularly when analyzing audio data in real time. This generative AI model analyzes the user's voice and provides evaluations based on pitch and rhythm.

[0072] The terminal provides an interface for users to access the system through a virtual reality device. This terminal has the function of capturing user actions and reflecting them in the virtual reality space. It also visually displays feedback based on voice analysis results, informing the user in real time. A distinctive feature of this system is that the terminal immediately reflects feedback, assisting in improving user performance.

[0073] After logging into the system, users can select a song from the provided music library and begin performing in a virtual reality space. Users receive feedback from their device and adjust their singing in real time, allowing them to improve their skills.

[0074] A concrete example would be a prompt message such as, "Provide a 3D model and effects based on the stage design selected by the user, and perform real-time voice analysis." In this way, each element works together to provide an intuitive and effective experience for the user.

[0075] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0076] Step 1:

[0077] The server receives the user's login information and generates a virtual reality space based on the selected theme. Once the login information and theme selection are entered, the server prepares 3D model data and effect data. For example, if the user selects "Rock Concert," the server generates a rock-themed stage design and prepares the necessary data.

[0078] Step 2:

[0079] The server streams the generated virtual reality data to the terminal. Here, 3D model data and effect data are input, and compressed data is output to the terminal. Specifically, a streaming protocol is used to efficiently deliver the data, allowing users to experience it without delay.

[0080] Step 3:

[0081] The device displays a virtual reality space to the user based on data received from the server. Compressed data from the server is the input, and the 3D space displayed on the virtual reality device is the output. Specifically, the avatar's movement on the stage is rendered in real time in response to the user's movements.

[0082] Step 4:

[0083] The device captures the user's movements and voice and sends them to the server. Here, user movement information and voice data are inputs, and raw data is output to the server. Specifically, data is collected using motion-tracking sensors and a microphone.

[0084] Step 5:

[0085] The server uses a generative AI model to analyze speech in real time and generate evaluation results. User speech data is the input, and the analyzed evaluation data is the output. Specifically, detailed analysis of pitch and rhythm is performed, and feedback information is generated.

[0086] Step 6:

[0087] The terminal receives feedback from the server and presents it to the user visually or audibly. The analyzed evaluation data is the input and becomes the visual or audible output to the user. Specifically, it displays off-key parts graphically and provides voice guidance indicating areas for improvement.

[0088] (Application Example 1)

[0089] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] The present invention aims to enable individual users to easily experience immersive virtual reality regardless of location, provide opportunities for improvement through real-time voice analysis and motion feedback, and create an environment that facilitates the sharing of experiences and exchange of feedback among users. In particular, it aims to enable a sufficiently immersive experience even in a home environment and improve access flexibility by utilizing smartphones.

[0091] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0092] In this invention, the server includes means for generating a virtual reality environment, means for detecting user actions and reflecting them in the virtual reality space, and intelligent processing means for analyzing and evaluating the user's voice in real time. This enables the provision of a high-quality immersive experience through a home environment or smartphone, and by utilizing voice analysis and feedback functions, it is possible to improve user performance and expand the experience.

[0093] A "virtual reality environment" is a digital world created by a computer in a 3D space that users can immerse themselves in and experience.

[0094] "Means for detecting user actions" refers to technologies that sense the movements of a user's body or device in real time and reflect them in the digital space.

[0095] "Intelligent processing means" refers to information processing technology that has the ability to analyze data such as user voice and make judgments and evaluations.

[0096] An "immersive experience" refers to an experience in which the user forgets physical reality and enters a virtual reality world, utilizing all senses such as sight and hearing.

[0097] "Feedback" is the process of providing users with evaluations and responses to their statements and actions, and it plays a role in encouraging improvement and new discoveries.

[0098] "Home environment" refers to the living space within a typical house, a place where daily use is expected.

[0099] A "smartphone" is a portable communication device that has communication capabilities and operates as a multi-functional computer.

[0100] To implement this invention, a system is built through the cooperation of a server, a terminal such as a smartphone, and a user. The server prepares the 3D spatial data and theme design necessary to generate the virtual reality environment and streams it to the terminal. To detect user movements, the terminal uses sensors from a smartphone or VR device to capture the user's movements in real time and reflect them in the virtual environment. In this process, 3D rendering is performed using software such as Unity or Unreal Engine.

[0101] Furthermore, the system utilizes speech recognition services such as Google® Cloud Speech-to-Text as an intelligent processing tool to analyze the user's speech in real time. The analysis results are provided to the user as visual or auditory feedback to encourage improvements in pitch and rhythm. This entire process creates an immersive concert experience.

[0102] As a concrete example, when a user selects an "Anime Song Festival" theme and sings a song, a 3D stage provided by the server is displayed on the terminal. The user's voice data is instantly processed intelligently, and feedback based on pitch and volume appears on the screen along with visual effects. In this way, the user can improve their performance within the virtual environment.

[0103] An example of a prompt from the generating AI is, "Please tell me any particular vocal points or performance aspects you should pay attention to on the stage of the anime song festival you have chosen." Through this prompt, the user can receive more precise feedback and advice from the generating AI.

[0104] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0105] Step 1:

[0106] The user logs into their device and selects a theme for the virtual reality concert. The user's theme selection information is passed to the device as input and sent as a request to the server. The server generates the necessary 3D environment data and stage design based on the selected theme and streams it to the device. As output, the 3D model and effect data are presented to the user's device.

[0107] Step 2:

[0108] The terminal captures the user's movements in real time using the sensors of the VR device or smartphone worn by the user. The user's movement information is input to the terminal, and data processing is performed using Unity or Unreal Engine technologies to reflect the movements in real time within the virtual reality space. The reflected user movements are then displayed in 3D space as output.

[0109] Step 3:

[0110] The user sings a song of their choice, and the device collects the audio data through its microphone. The user's audio data is captured by the device as input and sent to a server. The server uses a speech recognition service such as Google Cloud Speech-to-Text to analyze the audio data in real time. The analysis results are generated as feedback data as output.

[0111] Step 4:

[0112] The server generates feedback on pitch and rhythm based on the analysis results and sends it to the terminal. The input is the feedback information generated by the analysis, provided from the server to the terminal. The feedback is presented visually as highlighting and sound effects, and is used as educational material for the user. The output is information about areas for improvement in their own performance.

[0113] Step 5:

[0114] The user improves their singing based on feedback and performs again. The improved singing, based on the feedback, is processed again by the system and saved for sharing and evaluation with other users. As input, the improved performance data is entered into the terminal. As output, the user's improved performance is saved and shared, and feedback from other users can be received.

[0115] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0116] The system of this invention tracks the user's movements and voice as they engage in a real-time concert experience in a virtual reality space, and provides feedback based on this data. Furthermore, by incorporating an emotion engine, the system recognizes the user's emotions and adjusts the feedback from an emotional perspective.

[0117] The server generates a virtual reality space and prepares stage designs and themes according to the user's selection. This allows users to express themselves in diverse stage settings. The server also receives the user's voice and movement data in real time and evaluates it using artificial intelligence and an emotion engine. Voice data is analyzed based on criteria such as pitch, rhythm, and volume, and the emotion engine determines the user's emotional state from the characteristics of the voice and movements.

[0118] The device captures the user's physical movements and reflects them in the virtual space. When the user performs a specific movement, that movement is reflected in real time on the user avatar on the stage, enabling interactive performances. The device also receives evaluations and emotional feedback sent from the server and presents it to the user. The feedback is displayed as visual UI changes and audio advice.

[0119] Users can receive real-time emotional and technical feedback through their performance. For example, if a user sings a ballad with great emotion, the system will detect that emotion and provide more detailed expression guidance and emotional advice. This allows users to improve not only their technical skills but also their emotional expression. Furthermore, users can record their performances and share them with others, broadening the scope of feedback and deepening their interaction with the community.

[0120] For example, when a user sings an upbeat pop song, the emotion engine analyzes the excitement in the user's voice and the energy of their movements. If the user appears to be enjoying themselves, the feedback provides know-how to further enhance that enjoyment, and the virtual audience's reaction also becomes positive. In this way, the system provides comprehensive support that takes the user's emotions into account.

[0121] The following describes the processing flow.

[0122] Step 1:

[0123] The user logs into the system and puts on a virtual reality device. The device collects the user's authentication information and sends it to the server. The server authenticates the user's account based on this information and starts a session.

[0124] Step 2:

[0125] The user selects the stage design and theme for the virtual concert. The server prepares the corresponding 3D models and effects based on the user's selection and streams them to the device.

[0126] Step 3:

[0127] The device tracks the user's movements through the VR device and reflects them in the virtual reality space in real time. This allows the user's movements to be represented on a virtual stage.

[0128] Step 4:

[0129] The user selects a song and begins singing. The device collects the user's voice through the microphone and sends the audio data to the server.

[0130] Step 5:

[0131] The server uses artificial intelligence to analyze the user's voice data, evaluating pitch, rhythm, and volume. Simultaneously, an emotion engine performs sentiment analysis based on the user's voice tone and speed.

[0132] Step 6:

[0133] Based on the analysis results, the server generates technical and emotional feedback. This feedback includes suggestions for improving pitch and enhancing expressiveness.

[0134] Step 7:

[0135] The terminal receives feedback from the server and presents it to the user. Visually, this is done using on-screen instructions and color indicators, while auditorily, it is done using voice messages and guides.

[0136] Step 8:

[0137] Users continue singing while viewing and listening to feedback. Based on emotional feedback, users adjust their expression, aiming for a more emotionally rich performance.

[0138] Step 9:

[0139] After a singing session ends, users can save their performance data and share it with the community. The server accepts this data and makes it public so other users can access it.

[0140] Step 10:

[0141] Other users can watch shared performances and post reactions and comments. The server notifies the original user of these new interactions to encourage further feedback.

[0142] (Example 2)

[0143] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0144] Traditional virtual reality systems have limitations in tracking and evaluating user actions and voice in real time, making it difficult to accurately capture emotions and thus hindering improvements in the user experience. Furthermore, there are challenges in effectively sharing generated content and obtaining sufficient feedback from other users.

[0145] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0146] In this invention, the server includes means for generating a virtual reality environment, means for tracking the user's actions and reflecting them in the virtual space, and intelligent processing means for analyzing the user's voice and evaluating it in real time. This enables the provision of feedback based on advanced emotion recognition in real time and the efficient sharing of generated content.

[0147] A "virtual reality environment" is a three-dimensional virtual space created by a computer, an environment in which users can have an immersive experience.

[0148] "Tracking user movements" refers to the process of detecting and recording a user's physical movements using devices such as cameras and sensors.

[0149] "Reflecting in a virtual space" refers to the procedure of displaying or embodiing user actions and choices in a virtual reality environment in real time.

[0150] "Analyzing speech" refers to the process of using speech recognition technology to capture user speech data and extract its characteristics and meaning.

[0151] "Intelligent processing means" refers to a system or method for performing certain data processing, analysis, and evaluation using artificial intelligence.

[0152] "Recognizing emotional states" refers to methods of detecting or estimating a user's psychological or emotional state through their voice and actions.

[0153] "Adjusting feedback" refers to dynamically changing the content of the information and advice provided based on the user's situation and evaluation data.

[0154] "Recording generated information" refers to the process of saving digital data created or implemented by the user so that it can be used or shared later.

[0155] "Receiving feedback from other users" refers to the process of receiving ratings and comments from different users via the cloud or network.

[0156] This invention is a system for users to obtain an interactive experience in a virtual reality space. The embodiments thereof are described below.

[0157] First, the server is responsible for generating the virtual reality environment. Possible platforms for this include game engines such as Unity and Unreal Engine. These engines are used to build themes and stage designs in real time based on user selections. The server also incorporates intelligent processing capabilities to receive and analyze audio and motion data transmitted from the user. For audio analysis, it uses an audio processing library such as Librosa to extract audio characteristics. Furthermore, it uses an emotion analysis API (e.g., a natural language processing service) to evaluate the user's emotional state.

[0158] Next, the terminal plays a role in collecting the user's movements and voice. This is achieved using a VR headset or motion capture device (e.g., Kinect, Leap Motion). The data acquired from the terminal is reflected in the user's avatar in the virtual space, enabling dynamic interaction. The terminal also provides the user with visual and auditory feedback based on evaluation results sent from the server. This allows the user to obtain information that helps improve their performance in real time.

[0159] Finally, users can adjust and improve their performance based on real-time feedback. For example, if a user sings emotionally in a virtual concert, the system can detect that emotion and provide precise expression guidance and emotional advice. Users can also record their performances and share them with other users to receive more diverse feedback and deepen communication.

[0160] For example, when a user sings an upbeat pop song, the excitement in their voice and the energy of their movements are analyzed. If the user appears to be enjoying themselves, feedback is provided on how to further enhance that enjoyment, and the virtual audience's reaction is also designed to be positive.

[0161] As an example of a prompt, the generative AI model can be used with the following prompt: "Explain how to analyze a user's emotions and provide feedback when they sing an upbeat pop song in a virtual reality space." This allows the entire system to significantly improve the user's immersive experience.

[0162] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0163] Step 1:

[0164] The user puts on a VR headset and motion capture device and begins the virtual reality experience. The user's account information and initial settings are used as input. The connection to the virtual reality environment is configured as output.

[0165] Step 2:

[0166] The server generates a virtual reality environment based on user selections. The input includes settings such as the user's chosen stage theme and design. The output is a virtual space constructed using Unity or Unreal Engine.

[0167] Step 3:

[0168] The terminal captures the user's movements using a motion capture device. The user's physical movement data is acquired as input. The captured movement data is sent to the server as output and reflected in the avatar within the virtual space.

[0169] Step 4:

[0170] The device collects the user's voice through the microphone. The user's voice data is acquired as input. The collected voice data is sent to the server as output.

[0171] Step 5:

[0172] The server analyzes the received audio data. As input, the audio data is analyzed using an audio processing library such as Librosa. Data processing extracts characteristics such as pitch, rhythm, and volume. The analyzed audio characteristics data is obtained as output.

[0173] Step 6:

[0174] The server uses an emotion analysis API to evaluate the user's emotional state. Voice characteristic data and behavioral data are used as input. Emotion is estimated from voice tone and behavioral dynamics as part of the data calculation. Emotional evaluation data is generated as output.

[0175] Step 7:

[0176] The server generates feedback based on sentiment evaluation data. Speech characteristic data and sentiment evaluation data are used as input. Visual or auditory feedback information is created as output and sent to the terminal.

[0177] Step 8:

[0178] The terminal displays feedback received from the server to the user. Feedback information is acquired as input. UI changes and voice advice within the virtual space are displayed as output.

[0179] Step 9:

[0180] The user adjusts their performance based on the feedback provided. Feedback information is used as input. The output is an improvement in the user's performance, both emotionally and technically.

[0181] Step 10:

[0182] Users can record their performance and share it with other users. Audio and motion data from the session are used as input. As output, the generated content is saved to the cloud and can be shared with other participants.

[0183] (Application Example 2)

[0184] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0185] Traditional virtual reality technologies have made it difficult for users to receive emotionally stimulating feedback in real time to improve their performance. Furthermore, they have been unable to replicate realistic reactions from other users in a virtual space, resulting in a lack of immersion and interactivity. This has limited opportunities for users to proactively interact and enhance their self-expression.

[0186] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0187] In this invention, the server includes means for generating a virtual reality space, means for capturing user actions and reflecting them in the virtual reality space, and knowledge processing means for recognizing and evaluating the user's voice in real time. This allows the user to receive feedback that takes into account not only technical evaluations but also emotional aspects, realizing a highly immersive and interactive experience.

[0188] A "virtual reality space" is a three-dimensional virtual environment created using digital technology, an artificial space for users to have an immersive experience.

[0189] "Means for capturing user actions" refer to technologies and devices that detect the user's body movements in real time and analyze them as digital data.

[0190] A "knowledge processing tool" is a process that uses artificial intelligence technology to analyze user voice and actions and provide evaluations and feedback.

[0191] The "emotional aspect" refers to elements related to the user's emotions and mental state, and is the aspect that influences feedback and interaction.

[0192] "Providing appearances or stage designs according to user selection" means automatically setting visual themes and stage configurations in a virtual reality space based on the user's preferences and tastes.

[0193] "Virtually recreating other users' reactions" means providing an interactive environment by visually or audibly representing reactions that other users have never actually made in reality.

[0194] The system for implementing this invention provides a user performance experience using a virtual reality space. The server generates a virtual reality space and captures the user's actions and voice in real time within this space. This uses the camera and microphone installed on the user's device. The captured action data and voice data are transferred to the server and analyzed using artificial intelligence technology, which is a knowledge processing means. Software used here includes Google Cloud Speech-to-Text API and Microsoft® Azure® Text Analytics API.

[0195] The server sets up visual or auditory feedback based on the analysis results and sends it to the user in real time. An emotion engine is also used to determine the user's emotional state, and the feedback is adjusted from an emotional perspective. In this process, the setting or stage design is provided according to the user's choices, and immersion is enhanced by virtually recreating the reactions of other users.

[0196] As a concrete example, when a user performs a jazz piece, they can receive positive feedback from other users acting as a virtual audience. Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and movements during the performance and suggests more effective ways to express themselves. An example of a prompt might be, "By performing an upbeat jazz piece with rich emotion, what kind of emotions can you convey to the audience? Please provide some feedback examples."

[0197] The device provides users with visual and auditory feedback based on data received from the server. This allows users to attempt self-expression and see the results in real time. The invention enables users to improve their technical and emotional performance and enjoy interactive communication with other users.

[0198] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0199] Step 1:

[0200] The user logs into the virtual reality space using a terminal. Inputs include a user ID and password, and a selection of a virtual stage is also made. The output is initial setup information based on the selected stage. At this stage, the user reviews various settings and prepares for performance.

[0201] Step 2:

[0202] The device uses a camera and microphone to capture the user's actions and voice in real time. Input includes user movement data and voice data, which are sent to the server. Output is digitized movement data and voice data. User movements are captured by the camera, and voice is recorded by the microphone.

[0203] Step 3:

[0204] The server analyzes the transmitted data using knowledge processing tools. Inputs include behavioral and audio data, from which pitch, rhythm, and emotional information are analyzed. Outputs include evaluation results and emotional feedback. The data is analyzed using the Google Cloud Speech-to-Text API for speech analysis and the Microsoft Azure Text Analytics API for emotional analysis.

[0205] Step 4:

[0206] The server generates and sends feedback to the terminal based on the analysis results. The inputs used are the analyzed evaluation results and sentiment information. The output is feedback data delivered visually and aurally. The feedback is visualized as UI changes and voice advice, and actions are taken to enhance the user experience.

[0207] Step 5:

[0208] The device presents feedback received from the server to the user visually and audibly. Feedback data is used as input data. The output consists of messages of improvement and praise for the user. In this process, the device uses its display and speakers to deliver information and support the user's actions to learn from and enjoy the experience.

[0209] Step 6:

[0210] Based on the feedback received through their performance, users make adjustments and preparations for their next performance. The input is the content of the feedback, and the output is the performance plan that the user uses to make adjustments. Users analyze the received information and set goals for the next stage.

[0211] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0212] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0213] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0214] [Second Embodiment]

[0215] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0216] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0217] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0218] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0219] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0220] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0221] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0222] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0223] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0224] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0225] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0226] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0227] The system of the present invention is configured to enable users to experience an immersive concert in a virtual reality space. This system functions through the cooperation of a server, a terminal, and the user. The following describes each element and specific embodiments.

[0228] The server is responsible for generating the virtual reality environment and constructing themes and stage designs according to the user's selections. When a user chooses a specific stage design, the server prepares 3D models and effect data corresponding to that design and streams them to the terminal. At this time, the server receives the user's voice data and performs real-time voice analysis using artificial intelligence to evaluate the singing.

[0229] The terminal allows users to access the system using a virtual reality device and interact with it in a natural way. The terminal has the capability to capture user movements and reflect them in the virtual space. It also receives feedback based on user performance evaluations sent from the server and presents it to the user in an appropriate format. For example, if a user sings off-key, that part is visually highlighted.

[0230] After logging into the system, users can realize their performance in a virtual reality space while making various settings. Users can select songs from the content library provided by the system and start singing. They can also use their devices to receive real-time feedback, adjust their singing, and strive to improve their skills.

[0231] As a concrete example, suppose the user selects a rock concert stage. In this case, the server generates a rock-themed stage set, and the device reflects the user's movements in real time. When the user sings a song, artificial intelligence analyzes the voice in real time and provides accurate feedback. Based on this feedback, the user improves their singing.

[0232] Furthermore, this system also includes a feature that allows users to share the performance they generate with other users. The server stores user performance data and makes it available for sharing within the community. This expands opportunities for users to express themselves through feedback and evaluations from other users.

[0233] The following describes the processing flow.

[0234] Step 1:

[0235] The user logs into the system. The terminal obtains the user's authentication information from the input screen and sends it to the server. The server uses the received authentication information to refer to the database and authenticate the user's account.

[0236] Step 2:

[0237] The user selects a theme and stage for the virtual concert. Based on the user's selection, the server prepares the corresponding 3D models and effect data and gets ready to transfer them to the terminal.

[0238] Step 3:

[0239] The device initializes the VR device and tracks the user's position and movements in real time. This ensures that the user's movements are accurately reflected in the virtual reality space.

[0240] Step 4:

[0241] The user selects a song to sing. The device requests the selected song data from the server and stores the data in the cache as needed.

[0242] Step 5:

[0243] When the user starts singing, the device captures the user's voice from the microphone and streams that audio data to the server in real time.

[0244] Step 6:

[0245] The server uses artificial intelligence to analyze the received audio data and evaluates the user's singing based on various evaluation criteria such as pitch, rhythm, and volume.

[0246] Step 7:

[0247] The server generates evaluation results and sends them to the terminal as feedback. The terminal then presents this feedback to the user visually or audibly.

[0248] Step 8:

[0249] The user adjusts their singing based on the feedback and practices further. If necessary, the device records the user's performance and saves it for later playback.

[0250] Step 9:

[0251] If a user chooses to share their performance with the community, the server stores that data and makes it accessible to other users.

[0252] Step 10:

[0253] Other users can view the shared performance and leave comments and ratings. The server notifies the user who originally posted the performance of any newly added comments or ratings.

[0254] (Example 1)

[0255] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0256] Current virtual reality technologies make it difficult for users to experience immersive, real-time performances. Furthermore, there is a lack of immediate feedback to effectively promote individual user skill improvement. Additionally, there are limited means for sharing generated performances with others and stimulating communication.

[0257] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0258] In this invention, the server includes means having the function of generating a virtual reality space, means having the function of recording the user's activities and reflecting them in the virtual environment space, and an artificial intelligence device that recognizes the user's voice and performs analysis and evaluation in real time. As a result, the user can enjoy an immersive performance experience in real time, improve their skills by receiving immediate feedback, and effectively share the generated performance with other users.

[0259] A "virtual reality space" is a three-dimensional environment generated by a computer, in which users can have an immersive experience in real time.

[0260] An "information processing device" is an electronic device that performs data calculations, analysis, streaming, and other processing to enable virtual reality experiences.

[0261] "User activity" refers to the collective actions and voice inputs performed by the user within the virtual reality space, which are reflected in the virtual environment through the interface.

[0262] An "artificial intelligence device" is a set of software or hardware that analyzes user input such as voice and movement in real time through data analysis and learning, and provides evaluation and feedback.

[0263] "Subject" refers to a theme or concept in a virtual reality space, and is an element that expresses different styles and atmospheres that can be selected by the user.

[0264] "Spatial design" refers to the arrangement and design of environments and stages within a virtual reality space, and is an element that should be customized according to the user's preferences.

[0265] This invention is a system that enables users to conduct immersive performance experiences in a virtual reality space. Specific embodiments of each element are shown below.

[0266] The server plays a central role in generating the virtual reality space. Based on the theme selected by the user, the server prepares 3D designs and effects. For example, if the user chooses a rock concert theme, the server generates data for a rock-style stage set and streams it to the terminal. The server uses a generative AI model, particularly when analyzing audio data in real time. This generative AI model analyzes the user's voice and provides evaluations based on pitch and rhythm.

[0267] The terminal provides an interface for users to access the system through a virtual reality device. This terminal has the function of capturing user actions and reflecting them in the virtual reality space. It also visually displays feedback based on voice analysis results, informing the user in real time. A distinctive feature of this system is that the terminal immediately reflects feedback, assisting in improving user performance.

[0268] After logging into the system, users can select a song from the provided music library and begin performing in a virtual reality space. Users receive feedback from their device and adjust their singing in real time, allowing them to improve their skills.

[0269] A concrete example would be a prompt message such as, "Provide a 3D model and effects based on the stage design selected by the user, and perform real-time voice analysis." In this way, each element works together to provide an intuitive and effective experience for the user.

[0270] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0271] Step 1:

[0272] The server receives the user's login information and generates a virtual reality space based on the selected theme. Once the login information and theme selection are entered, the server prepares 3D model data and effect data. For example, if the user selects "Rock Concert," the server generates a rock-themed stage design and prepares the necessary data.

[0273] Step 2:

[0274] The server streams the generated virtual reality data to the terminal. Here, 3D model data and effect data are input, and compressed data is output to the terminal. Specifically, a streaming protocol is used to efficiently deliver the data, allowing users to experience it without delay.

[0275] Step 3:

[0276] The device displays a virtual reality space to the user based on data received from the server. Compressed data from the server is the input, and the 3D space displayed on the virtual reality device is the output. Specifically, the avatar's movement on the stage is rendered in real time in response to the user's movements.

[0277] Step 4:

[0278] The device captures the user's movements and voice and sends them to the server. Here, user movement information and voice data are inputs, and raw data is output to the server. Specifically, data is collected using motion-tracking sensors and a microphone.

[0279] Step 5:

[0280] The server uses a generative AI model to analyze speech in real time and generate evaluation results. User speech data is the input, and the analyzed evaluation data is the output. Specifically, detailed analysis of pitch and rhythm is performed, and feedback information is generated.

[0281] Step 6:

[0282] The terminal receives feedback from the server and presents it visually or aurally to the user. The analyzed evaluation data is the input and becomes the visual or aural output to the user. As a specific operation, the part where the pitch is off is displayed graphically, and the improvement points are notified by voice guidance.

[0283] (Application Example 1)

[0284] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0285] The present invention aims to enable each individual user to easily have an immersive experience without depending on location in a conventional virtual reality system, further provide improvement opportunities through real-time voice analysis and reflection of actions, and provide an environment where sharing of experiences and exchange of feedback between users can be smoothly performed. In particular, it is an issue to enable a sufficient immersive experience even in a home environment and improve the flexibility of access by using a smartphone.

[0286] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0287] In this invention, the server includes means for generating a virtual reality environment, means for detecting the user's actions and reflecting them in the virtual reality space, and intelligent processing means for analyzing the user's voice and evaluating it in real time. Thereby, a high-quality immersive experience is provided through a home environment or a smartphone, and by utilizing the voice analysis and feedback functions, it becomes possible to improve the user's performance and expand the experience.

[0288] The "virtual reality environment" is a 3D space generated by a computer, a digital world in which a user can immerse and experience.

[0289] "Means for detecting user actions" refers to technologies that sense the movements of a user's body or device in real time and reflect them in the digital space.

[0290] "Intelligent processing means" refers to information processing technology that has the ability to analyze data such as user voice and make judgments and evaluations.

[0291] An "immersive experience" refers to an experience in which the user forgets physical reality and enters a virtual reality world, utilizing all senses such as sight and hearing.

[0292] "Feedback" is the process of providing users with evaluations and responses to their statements and actions, and it plays a role in encouraging improvement and new discoveries.

[0293] "Home environment" refers to the living space within a typical house, a place where daily use is expected.

[0294] A "smartphone" is a portable communication device that has communication capabilities and operates as a multi-functional computer.

[0295] To implement this invention, a system is built through the cooperation of a server, a terminal such as a smartphone, and a user. The server prepares the 3D spatial data and theme design necessary to generate the virtual reality environment and streams it to the terminal. To detect user movements, the terminal uses sensors from a smartphone or VR device to capture the user's movements in real time and reflect them in the virtual environment. In this process, 3D rendering is performed using software such as Unity or Unreal Engine.

[0296] Furthermore, the system utilizes speech recognition services such as Google Cloud Speech-to-Text as an intelligent processing tool to analyze the user's speech in real time. The analysis results are provided to the user as visual or auditory feedback, encouraging improvements in pitch and rhythm. This entire process creates an immersive concert experience.

[0297] As a concrete example, when a user selects an "Anime Song Festival" theme and sings a song, a 3D stage provided by the server is displayed on the terminal. The user's voice data is instantly processed intelligently, and feedback based on pitch and volume appears on the screen along with visual effects. In this way, the user can improve their performance within the virtual environment.

[0298] An example of a prompt from the generating AI is, "Please tell me any particular vocal points or performance aspects you should pay attention to on the stage of the anime song festival you have chosen." Through this prompt, the user can receive more precise feedback and advice from the generating AI.

[0299] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0300] Step 1:

[0301] The user logs into their device and selects a theme for the virtual reality concert. The user's theme selection information is passed to the device as input and sent as a request to the server. The server generates the necessary 3D environment data and stage design based on the selected theme and streams it to the device. As output, the 3D model and effect data are presented to the user's device.

[0302] Step 2:

[0303] The terminal captures the user's movements in real time using the sensors of the VR device or smartphone worn by the user. As input, the user's movement information is input into the terminal, and data processing is performed to reflect the movement in real time within the virtual reality space using technologies such as Unity and Unreal Engine. As output, the reflected user movement is displayed in the 3D space.

[0304] Step 3:

[0305] The user sings the selected song, and the terminal collects the voice data through the microphone. As input, the user's voice data is captured by the terminal and sent to the server. The server uses a speech recognition service such as Google Cloud Speech-to-Text to analyze the voice data in real time. As output, the analysis result is generated as feedback data.

[0306] Step 4:

[0307] Based on the analysis result, the server generates feedback regarding the pitch and rhythm and sends it to the terminal. As input, the feedback information from the analysis is provided by the server to the terminal. The feedback is presented visually as highlights or as sound effects and is utilized as a teaching material for the user. As output, the user obtains information on areas for improvement regarding their performance.

[0308] Step 5:

[0309] The user improves their singing based on the feedback and performs again. The improved singing based on the feedback is processed again by the system and saved for sharing and evaluation with other users. As input, the performance data after improvement is input into the terminal. As output, the user's improved performance is saved and shared, and it becomes possible to receive feedback from other users.

[0310] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0311] The system of this invention tracks the user's movements and voice as they engage in a real-time concert experience in a virtual reality space, and provides feedback based on this data. Furthermore, by incorporating an emotion engine, the system recognizes the user's emotions and adjusts the feedback from an emotional perspective.

[0312] The server generates a virtual reality space and prepares stage designs and themes according to the user's selection. This allows users to express themselves in diverse stage settings. The server also receives the user's voice and movement data in real time and evaluates it using artificial intelligence and an emotion engine. Voice data is analyzed based on criteria such as pitch, rhythm, and volume, and the emotion engine determines the user's emotional state from the characteristics of the voice and movements.

[0313] The device captures the user's physical movements and reflects them in the virtual space. When the user performs a specific movement, that movement is reflected in real time on the user avatar on the stage, enabling interactive performances. The device also receives evaluations and emotional feedback sent from the server and presents it to the user. The feedback is displayed as visual UI changes and audio advice.

[0314] Users can receive real-time emotional and technical feedback through their performance. For example, if a user sings a ballad with great emotion, the system will detect that emotion and provide more detailed expression guidance and emotional advice. This allows users to improve not only their technical skills but also their emotional expression. Furthermore, users can record their performances and share them with others, broadening the scope of feedback and deepening their interaction with the community.

[0315] For example, when a user sings an upbeat pop song, the emotion engine analyzes the excitement in the user's voice and the energy of their movements. If the user appears to be enjoying themselves, the feedback provides know-how to further enhance that enjoyment, and the virtual audience's reaction also becomes positive. In this way, the system provides comprehensive support that takes the user's emotions into account.

[0316] The following describes the processing flow.

[0317] Step 1:

[0318] The user logs into the system and puts on a virtual reality device. The device collects the user's authentication information and sends it to the server. The server authenticates the user's account based on this information and starts a session.

[0319] Step 2:

[0320] The user selects the stage design and theme for the virtual concert. The server prepares the corresponding 3D models and effects based on the user's selection and streams them to the device.

[0321] Step 3:

[0322] The device tracks the user's movements through the VR device and reflects them in the virtual reality space in real time. This allows the user's movements to be represented on a virtual stage.

[0323] Step 4:

[0324] The user selects a song and begins singing. The device collects the user's voice through the microphone and sends the audio data to the server.

[0325] Step 5:

[0326] The server uses artificial intelligence to analyze the user's voice data, evaluating pitch, rhythm, and volume. Simultaneously, an emotion engine performs sentiment analysis based on the user's voice tone and speed.

[0327] Step 6:

[0328] Based on the analysis results, the server generates technical and emotional feedback. This feedback includes suggestions for improving pitch and enhancing expressiveness.

[0329] Step 7:

[0330] The terminal receives feedback from the server and presents it to the user. Visually, this is done using on-screen instructions and color indicators, while auditorily, it is done using voice messages and guides.

[0331] Step 8:

[0332] Users continue singing while viewing and listening to feedback. Based on emotional feedback, users adjust their expression, aiming for a more emotionally rich performance.

[0333] Step 9:

[0334] After a singing session ends, users can save their performance data and share it with the community. The server accepts this data and makes it public so other users can access it.

[0335] Step 10:

[0336] Other users can watch shared performances and post reactions and comments. The server notifies the original user of these new interactions to encourage further feedback.

[0337] (Example 2)

[0338] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0339] Traditional virtual reality systems have limitations in tracking and evaluating user actions and voice in real time, making it difficult to accurately capture emotions and thus hindering improvements in the user experience. Furthermore, there are challenges in effectively sharing generated content and obtaining sufficient feedback from other users.

[0340] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0341] In this invention, the server includes means for generating a virtual reality environment, means for tracking the user's actions and reflecting them in the virtual space, and intelligent processing means for analyzing the user's voice and evaluating it in real time. This enables the provision of feedback based on advanced emotion recognition in real time and the efficient sharing of generated content.

[0342] A "virtual reality environment" is a three-dimensional virtual space created by a computer, an environment in which users can have an immersive experience.

[0343] "Tracking user movements" refers to the process of detecting and recording a user's physical movements using devices such as cameras and sensors.

[0344] "Reflecting in a virtual space" refers to the procedure of displaying or embodiing user actions and choices in a virtual reality environment in real time.

[0345] "Analyzing speech" refers to the process of using speech recognition technology to capture user speech data and extract its characteristics and meaning.

[0346] "Intelligent processing means" refers to a system or method for performing certain data processing, analysis, and evaluation using artificial intelligence.

[0347] "Recognizing emotional states" refers to methods of detecting or estimating a user's psychological or emotional state through their voice and actions.

[0348] "Adjusting feedback" refers to dynamically changing the content of the information and advice provided based on the user's situation and evaluation data.

[0349] "Recording generated information" refers to the process of saving digital data created or implemented by the user so that it can be used or shared later.

[0350] "Receiving feedback from other users" refers to the process of receiving ratings and comments from different users via the cloud or network.

[0351] This invention is a system for users to obtain an interactive experience in a virtual reality space. The embodiments thereof are described below.

[0352] First, the server is responsible for generating the virtual reality environment. Possible platforms for this include game engines such as Unity and Unreal Engine. These engines are used to build themes and stage designs in real time based on user selections. The server also incorporates intelligent processing capabilities to receive and analyze audio and motion data transmitted from the user. For audio analysis, it uses an audio processing library such as Librosa to extract audio characteristics. Furthermore, it uses an emotion analysis API (e.g., a natural language processing service) to evaluate the user's emotional state.

[0353] Next, the terminal plays a role in collecting the user's movements and voice. This is achieved using a VR headset or motion capture device (e.g., Kinect, Leap Motion). The data acquired from the terminal is reflected in the user's avatar in the virtual space, enabling dynamic interaction. The terminal also provides the user with visual and auditory feedback based on evaluation results sent from the server. This allows the user to obtain information that helps improve their performance in real time.

[0354] Finally, users can adjust and improve their performance based on real-time feedback. For example, if a user sings emotionally in a virtual concert, the system can detect that emotion and provide precise expression guidance and emotional advice. Users can also record their performances and share them with other users to receive more diverse feedback and deepen communication.

[0355] For example, when a user sings an upbeat pop song, the excitement in their voice and the energy of their movements are analyzed. If the user appears to be enjoying themselves, feedback is provided on how to further enhance that enjoyment, and the virtual audience's reaction is also designed to be positive.

[0356] As an example of a prompt, the generative AI model can be used with the following prompt: "Explain how to analyze a user's emotions and provide feedback when they sing an upbeat pop song in a virtual reality space." This allows the entire system to significantly improve the user's immersive experience.

[0357] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0358] Step 1:

[0359] The user puts on a VR headset and motion capture device and begins the virtual reality experience. The user's account information and initial settings are used as input. The connection to the virtual reality environment is configured as output.

[0360] Step 2:

[0361] The server generates a virtual reality environment based on user selections. The input includes settings such as the user's chosen stage theme and design. The output is a virtual space constructed using Unity or Unreal Engine.

[0362] Step 3:

[0363] The terminal captures the user's movements using a motion capture device. The user's physical movement data is acquired as input. The captured movement data is sent to the server as output and reflected in the avatar within the virtual space.

[0364] Step 4:

[0365] The device collects the user's voice through the microphone. The user's voice data is acquired as input. The collected voice data is sent to the server as output.

[0366] Step 5:

[0367] The server analyzes the received audio data. As input, the audio data is analyzed using an audio processing library such as Librosa. Data processing extracts characteristics such as pitch, rhythm, and volume. The analyzed audio characteristics data is obtained as output.

[0368] Step 6:

[0369] The server uses an emotion analysis API to evaluate the user's emotional state. Voice characteristic data and behavioral data are used as input. Emotion is estimated from voice tone and behavioral dynamics as part of the data calculation. Emotional evaluation data is generated as output.

[0370] Step 7:

[0371] The server generates feedback based on sentiment evaluation data. Speech characteristic data and sentiment evaluation data are used as input. Visual or auditory feedback information is created as output and sent to the terminal.

[0372] Step 8:

[0373] The terminal displays feedback received from the server to the user. Feedback information is acquired as input. UI changes and voice advice within the virtual space are displayed as output.

[0374] Step 9:

[0375] The user adjusts their performance based on the feedback provided. Feedback information is used as input. The output is an improvement in the user's performance, both emotionally and technically.

[0376] Step 10:

[0377] Users can record their performance and share it with other users. Audio and motion data from the session are used as input. As output, the generated content is saved to the cloud and can be shared with other participants.

[0378] (Application Example 2)

[0379] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0380] Traditional virtual reality technologies have made it difficult for users to receive emotionally stimulating feedback in real time to improve their performance. Furthermore, they have been unable to replicate realistic reactions from other users in a virtual space, resulting in a lack of immersion and interactivity. This has limited opportunities for users to proactively interact and enhance their self-expression.

[0381] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0382] In this invention, the server includes means for generating a virtual reality space, means for capturing user actions and reflecting them in the virtual reality space, and knowledge processing means for recognizing and evaluating the user's voice in real time. This allows the user to receive feedback that takes into account not only technical evaluations but also emotional aspects, realizing a highly immersive and interactive experience.

[0383] A "virtual reality space" is a three-dimensional virtual environment created using digital technology, an artificial space for users to have an immersive experience.

[0384] "Means for capturing user actions" refer to technologies and devices that detect the user's body movements in real time and analyze them as digital data.

[0385] A "knowledge processing tool" is a process that uses artificial intelligence technology to analyze user voice and actions and provide evaluations and feedback.

[0386] The "emotional aspect" refers to elements related to the user's emotions and mental state, and is the aspect that influences feedback and interaction.

[0387] "Providing appearances or stage designs according to user selection" means automatically setting visual themes and stage configurations in a virtual reality space based on the user's preferences and tastes.

[0388] "Virtually recreating other users' reactions" means providing an interactive environment by visually or audibly representing reactions that other users have never actually made in reality.

[0389] The system for implementing this invention provides a user performance experience using a virtual reality space. A server generates a virtual reality space and captures the user's actions and voice in real time within this space. This uses a camera and microphone installed on the user's device. The captured action and voice data are transferred to the server and analyzed using artificial intelligence technology, which is a knowledge processing tool. Software used in this system includes Google Cloud Speech-to-Text API and Microsoft Azure Text Analytics API.

[0390] The server sets up visual or auditory feedback based on the analysis results and sends it to the user in real time. An emotion engine is also used to determine the user's emotional state, and the feedback is adjusted from an emotional perspective. In this process, the setting or stage design is provided according to the user's choices, and immersion is enhanced by virtually recreating the reactions of other users.

[0391] As a concrete example, when a user performs a jazz piece, they can receive positive feedback from other users acting as a virtual audience. Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and movements during the performance and suggests more effective ways to express themselves. An example of a prompt might be, "By performing an upbeat jazz piece with rich emotion, what kind of emotions can you convey to the audience? Please provide some feedback examples."

[0392] The device provides users with visual and auditory feedback based on data received from the server. This allows users to attempt self-expression and see the results in real time. The invention enables users to improve their technical and emotional performance and enjoy interactive communication with other users.

[0393] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0394] Step 1:

[0395] The user logs into the virtual reality space using a terminal. Inputs include a user ID and password, and a selection of a virtual stage is also made. The output is initial setup information based on the selected stage. At this stage, the user reviews various settings and prepares for performance.

[0396] Step 2:

[0397] The device uses a camera and microphone to capture the user's actions and voice in real time. Input includes user movement data and voice data, which are sent to the server. Output is digitized movement data and voice data. User movements are captured by the camera, and voice is recorded by the microphone.

[0398] Step 3:

[0399] The server analyzes the transmitted data using knowledge processing tools. Inputs include behavioral and audio data, from which pitch, rhythm, and emotional information are analyzed. Outputs include evaluation results and emotional feedback. The data is analyzed using the Google Cloud Speech-to-Text API for speech analysis and the Microsoft Azure Text Analytics API for emotional analysis.

[0400] Step 4:

[0401] The server generates and sends feedback to the terminal based on the analysis results. The inputs used are the analyzed evaluation results and sentiment information. The output is feedback data delivered visually and aurally. The feedback is visualized as UI changes and voice advice, and actions are taken to enhance the user experience.

[0402] Step 5:

[0403] The device presents feedback received from the server to the user visually and audibly. Feedback data is used as input data. The output consists of messages of improvement and praise for the user. In this process, the device uses its display and speakers to deliver information and support the user's actions to learn from and enjoy the experience.

[0404] Step 6:

[0405] Based on the feedback received through their performance, users make adjustments and preparations for their next performance. The input is the content of the feedback, and the output is the performance plan that the user uses to make adjustments. Users analyze the received information and set goals for the next stage.

[0406] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0407] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0408] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0409] [Third Embodiment]

[0410] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0411] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0412] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0413] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0414] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0415] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0416] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0417] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0418] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0419] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0420] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0421] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0422] The system of the present invention is configured to enable users to experience an immersive concert in a virtual reality space. This system functions through the cooperation of a server, a terminal, and the user. The following describes each element and specific embodiments.

[0423] The server is responsible for generating the virtual reality environment and constructing themes and stage designs according to the user's selections. When a user chooses a specific stage design, the server prepares 3D models and effect data corresponding to that design and streams them to the terminal. At this time, the server receives the user's voice data and performs real-time voice analysis using artificial intelligence to evaluate the singing.

[0424] The terminal allows users to access the system using a virtual reality device and interact with it in a natural way. The terminal has the capability to capture user movements and reflect them in the virtual space. It also receives feedback based on user performance evaluations sent from the server and presents it to the user in an appropriate format. For example, if a user sings off-key, that part is visually highlighted.

[0425] After logging into the system, users can realize their performance in a virtual reality space while making various settings. Users can select songs from the content library provided by the system and start singing. They can also use their devices to receive real-time feedback, adjust their singing, and strive to improve their skills.

[0426] As a concrete example, suppose the user selects a rock concert stage. In this case, the server generates a rock-themed stage set, and the device reflects the user's movements in real time. When the user sings a song, artificial intelligence analyzes the voice in real time and provides accurate feedback. Based on this feedback, the user improves their singing.

[0427] Furthermore, this system also includes a feature that allows users to share the performance they generate with other users. The server stores user performance data and makes it available for sharing within the community. This expands opportunities for users to express themselves through feedback and evaluations from other users.

[0428] The following describes the processing flow.

[0429] Step 1:

[0430] The user logs into the system. The terminal obtains the user's authentication information from the input screen and sends it to the server. The server uses the received authentication information to refer to the database and authenticate the user's account.

[0431] Step 2:

[0432] The user selects a theme and stage for the virtual concert. Based on the user's selection, the server prepares the corresponding 3D models and effect data and gets ready to transfer them to the terminal.

[0433] Step 3:

[0434] The device initializes the VR device and tracks the user's position and movements in real time. This ensures that the user's movements are accurately reflected in the virtual reality space.

[0435] Step 4:

[0436] The user selects a song to sing. The device requests the selected song data from the server and stores the data in the cache as needed.

[0437] Step 5:

[0438] When the user starts singing, the device captures the user's voice from the microphone and streams that audio data to the server in real time.

[0439] Step 6:

[0440] The server uses artificial intelligence to analyze the received audio data and evaluates the user's singing based on various evaluation criteria such as pitch, rhythm, and volume.

[0441] Step 7:

[0442] The server generates evaluation results and sends them to the terminal as feedback. The terminal then presents this feedback to the user visually or audibly.

[0443] Step 8:

[0444] The user adjusts their singing based on the feedback and practices further. If necessary, the device records the user's performance and saves it for later playback.

[0445] Step 9:

[0446] If a user chooses to share their performance with the community, the server stores that data and makes it accessible to other users.

[0447] Step 10:

[0448] Other users can view the shared performance and leave comments and ratings. The server notifies the user who originally posted the performance of any newly added comments or ratings.

[0449] (Example 1)

[0450] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0451] Current virtual reality technologies make it difficult for users to experience immersive, real-time performances. Furthermore, there is a lack of immediate feedback to effectively promote individual user skill improvement. Additionally, there are limited means for sharing generated performances with others and stimulating communication.

[0452] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0453] In this invention, the server includes means having the function of generating a virtual reality space, means having the function of recording the user's activities and reflecting them in the virtual environment space, and an artificial intelligence device that recognizes the user's voice and performs analysis and evaluation in real time. As a result, the user can enjoy an immersive performance experience in real time, improve their skills by receiving immediate feedback, and effectively share the generated performance with other users.

[0454] A "virtual reality space" is a three-dimensional environment generated by a computer, in which users can have an immersive experience in real time.

[0455] An "information processing device" is an electronic device that performs data calculations, analysis, streaming, and other processing to enable virtual reality experiences.

[0456] "User activity" refers to the collective actions and voice inputs performed by the user within the virtual reality space, which are reflected in the virtual environment through the interface.

[0457] An "artificial intelligence device" is a set of software or hardware that analyzes user input such as voice and movement in real time through data analysis and learning, and provides evaluation and feedback.

[0458] "Subject" refers to a theme or concept in a virtual reality space, and is an element that expresses different styles and atmospheres that can be selected by the user.

[0459] "Spatial design" refers to the arrangement and design of environments and stages within a virtual reality space, and is an element that should be customized according to the user's preferences.

[0460] This invention is a system that enables users to conduct immersive performance experiences in a virtual reality space. Specific embodiments of each element are shown below.

[0461] The server plays a central role in generating the virtual reality space. Based on the theme selected by the user, the server prepares 3D designs and effects. For example, if the user chooses a rock concert theme, the server generates data for a rock-style stage set and streams it to the terminal. The server uses a generative AI model, particularly when analyzing audio data in real time. This generative AI model analyzes the user's voice and provides evaluations based on pitch and rhythm.

[0462] The terminal provides an interface for users to access the system through a virtual reality device. This terminal has the function of capturing user actions and reflecting them in the virtual reality space. It also visually displays feedback based on voice analysis results, informing the user in real time. A distinctive feature of this system is that the terminal immediately reflects feedback, assisting in improving user performance.

[0463] After logging into the system, users can select a song from the provided music library and begin performing in a virtual reality space. Users receive feedback from their device and adjust their singing in real time, allowing them to improve their skills.

[0464] A concrete example would be a prompt message such as, "Provide a 3D model and effects based on the stage design selected by the user, and perform real-time voice analysis." In this way, each element works together to provide an intuitive and effective experience for the user.

[0465] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0466] Step 1:

[0467] The server receives the user's login information and generates a virtual reality space based on the selected theme. Once the login information and theme selection are entered, the server prepares 3D model data and effect data. For example, if the user selects "Rock Concert," the server generates a rock-themed stage design and prepares the necessary data.

[0468] Step 2:

[0469] The server streams the generated virtual reality data to the terminal. Here, 3D model data and effect data are input, and compressed data is output to the terminal. Specifically, a streaming protocol is used to efficiently deliver the data, allowing users to experience it without delay.

[0470] Step 3:

[0471] The device displays a virtual reality space to the user based on data received from the server. Compressed data from the server is the input, and the 3D space displayed on the virtual reality device is the output. Specifically, the avatar's movement on the stage is rendered in real time in response to the user's movements.

[0472] Step 4:

[0473] The device captures the user's movements and voice and sends them to the server. Here, user movement information and voice data are inputs, and raw data is output to the server. Specifically, data is collected using motion-tracking sensors and a microphone.

[0474] Step 5:

[0475] The server uses a generative AI model to analyze speech in real time and generate evaluation results. User speech data is the input, and the analyzed evaluation data is the output. Specifically, detailed analysis of pitch and rhythm is performed, and feedback information is generated.

[0476] Step 6:

[0477] The terminal receives feedback from the server and presents it to the user visually or audibly. The analyzed evaluation data is the input and becomes the visual or audible output to the user. Specifically, it displays off-key parts graphically and provides voice guidance indicating areas for improvement.

[0478] (Application Example 1)

[0479] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0480] The present invention aims to enable individual users to easily experience immersive virtual reality regardless of location, provide opportunities for improvement through real-time voice analysis and motion feedback, and create an environment that facilitates the sharing of experiences and exchange of feedback among users. In particular, it aims to enable a sufficiently immersive experience even in a home environment and improve access flexibility by utilizing smartphones.

[0481] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0482] In this invention, the server includes means for generating a virtual reality environment, means for detecting user actions and reflecting them in the virtual reality space, and intelligent processing means for analyzing and evaluating the user's voice in real time. This enables the provision of a high-quality immersive experience through a home environment or smartphone, and by utilizing voice analysis and feedback functions, it is possible to improve user performance and expand the experience.

[0483] A "virtual reality environment" is a digital world created by a computer in a 3D space that users can immerse themselves in and experience.

[0484] "Means for detecting user actions" refers to technologies that sense the movements of a user's body or device in real time and reflect them in the digital space.

[0485] "Intelligent processing means" refers to information processing technology that has the ability to analyze data such as user voice and make judgments and evaluations.

[0486] An "immersive experience" refers to an experience in which the user forgets physical reality and enters a virtual reality world, utilizing all senses such as sight and hearing.

[0487] "Feedback" is the process of providing users with evaluations and responses to their statements and actions, and it plays a role in encouraging improvement and new discoveries.

[0488] "Home environment" refers to the living space within a typical house, a place where daily use is expected.

[0489] A "smartphone" is a portable communication device that has communication capabilities and operates as a multi-functional computer.

[0490] To implement this invention, a system is built through the cooperation of a server, a terminal such as a smartphone, and a user. The server prepares the 3D spatial data and theme design necessary to generate the virtual reality environment and streams it to the terminal. To detect user movements, the terminal uses sensors from a smartphone or VR device to capture the user's movements in real time and reflect them in the virtual environment. In this process, 3D rendering is performed using software such as Unity or Unreal Engine.

[0491] Furthermore, the system utilizes speech recognition services such as Google Cloud Speech-to-Text as an intelligent processing tool to analyze the user's speech in real time. The analysis results are provided to the user as visual or auditory feedback, encouraging improvements in pitch and rhythm. This entire process creates an immersive concert experience.

[0492] As a concrete example, when a user selects an "Anime Song Festival" theme and sings a song, a 3D stage provided by the server is displayed on the terminal. The user's voice data is instantly processed intelligently, and feedback based on pitch and volume appears on the screen along with visual effects. In this way, the user can improve their performance within the virtual environment.

[0493] An example of a prompt from the generating AI is, "Please tell me any particular vocal points or performance aspects you should pay attention to on the stage of the anime song festival you have chosen." Through this prompt, the user can receive more precise feedback and advice from the generating AI.

[0494] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0495] Step 1:

[0496] The user logs into their device and selects a theme for the virtual reality concert. The user's theme selection information is passed to the device as input and sent as a request to the server. The server generates the necessary 3D environment data and stage design based on the selected theme and streams it to the device. As output, the 3D model and effect data are presented to the user's device.

[0497] Step 2:

[0498] The terminal captures the user's movements in real time using the sensors of the VR device or smartphone worn by the user. The user's movement information is input to the terminal, and data processing is performed using Unity or Unreal Engine technologies to reflect the movements in real time within the virtual reality space. The reflected user movements are then displayed in 3D space as output.

[0499] Step 3:

[0500] The user sings a song of their choice, and the device collects the audio data through its microphone. The user's audio data is captured by the device as input and sent to a server. The server uses a speech recognition service such as Google Cloud Speech-to-Text to analyze the audio data in real time. The analysis results are generated as feedback data as output.

[0501] Step 4:

[0502] The server generates feedback on pitch and rhythm based on the analysis results and sends it to the terminal. The input is the feedback information generated by the analysis, provided from the server to the terminal. The feedback is presented visually as highlighting and sound effects, and is used as educational material for the user. The output is information about areas for improvement in their own performance.

[0503] Step 5:

[0504] The user improves their singing based on feedback and performs again. The improved singing, based on the feedback, is processed again by the system and saved for sharing and evaluation with other users. As input, the improved performance data is entered into the terminal. As output, the user's improved performance is saved and shared, and feedback from other users can be received.

[0505] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0506] The system of this invention tracks the user's movements and voice as they engage in a real-time concert experience in a virtual reality space, and provides feedback based on this data. Furthermore, by incorporating an emotion engine, the system recognizes the user's emotions and adjusts the feedback from an emotional perspective.

[0507] The server generates a virtual reality space and prepares stage designs and themes according to the user's selection. This allows users to express themselves in diverse stage settings. The server also receives the user's voice and movement data in real time and evaluates it using artificial intelligence and an emotion engine. Voice data is analyzed based on criteria such as pitch, rhythm, and volume, and the emotion engine determines the user's emotional state from the characteristics of the voice and movements.

[0508] The device captures the user's physical movements and reflects them in the virtual space. When the user performs a specific movement, that movement is reflected in real time on the user avatar on the stage, enabling interactive performances. The device also receives evaluations and emotional feedback sent from the server and presents it to the user. The feedback is displayed as visual UI changes and audio advice.

[0509] Users can receive real-time emotional and technical feedback through their performance. For example, if a user sings a ballad with great emotion, the system will detect that emotion and provide more detailed expression guidance and emotional advice. This allows users to improve not only their technical skills but also their emotional expression. Furthermore, users can record their performances and share them with others, broadening the scope of feedback and deepening their interaction with the community.

[0510] For example, when a user sings an upbeat pop song, the emotion engine analyzes the excitement in the user's voice and the energy of their movements. If the user appears to be enjoying themselves, the feedback provides know-how to further enhance that enjoyment, and the virtual audience's reaction also becomes positive. In this way, the system provides comprehensive support that takes the user's emotions into account.

[0511] The following describes the processing flow.

[0512] Step 1:

[0513] The user logs into the system and puts on a virtual reality device. The device collects the user's authentication information and sends it to the server. The server authenticates the user's account based on this information and starts a session.

[0514] Step 2:

[0515] The user selects the stage design and theme for the virtual concert. The server prepares the corresponding 3D models and effects based on the user's selection and streams them to the device.

[0516] Step 3:

[0517] The device tracks the user's movements through the VR device and reflects them in the virtual reality space in real time. This allows the user's movements to be represented on a virtual stage.

[0518] Step 4:

[0519] The user selects a song and begins singing. The device collects the user's voice through the microphone and sends the audio data to the server.

[0520] Step 5:

[0521] The server uses artificial intelligence to analyze the user's voice data, evaluating pitch, rhythm, and volume. Simultaneously, an emotion engine performs sentiment analysis based on the user's voice tone and speed.

[0522] Step 6:

[0523] Based on the analysis results, the server generates technical and emotional feedback. This feedback includes suggestions for improving pitch and enhancing expressiveness.

[0524] Step 7:

[0525] The terminal receives feedback from the server and presents it to the user. Visually, this is done using on-screen instructions and color indicators, while auditorily, it is done using voice messages and guides.

[0526] Step 8:

[0527] Users continue singing while viewing and listening to feedback. Based on emotional feedback, users adjust their expression, aiming for a more emotionally rich performance.

[0528] Step 9:

[0529] After a singing session ends, users can save their performance data and share it with the community. The server accepts this data and makes it public so other users can access it.

[0530] Step 10:

[0531] Other users can watch shared performances and post reactions and comments. The server notifies the original user of these new interactions to encourage further feedback.

[0532] (Example 2)

[0533] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0534] Traditional virtual reality systems have limitations in tracking and evaluating user actions and voice in real time, making it difficult to accurately capture emotions and thus hindering improvements in the user experience. Furthermore, there are challenges in effectively sharing generated content and obtaining sufficient feedback from other users.

[0535] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0536] In this invention, the server includes means for generating a virtual reality environment, means for tracking the user's actions and reflecting them in the virtual space, and intelligent processing means for analyzing the user's voice and evaluating it in real time. This enables the provision of feedback based on advanced emotion recognition in real time and the efficient sharing of generated content.

[0537] A "virtual reality environment" is a three-dimensional virtual space created by a computer, an environment in which users can have an immersive experience.

[0538] "Tracking user movements" refers to the process of detecting and recording a user's physical movements using devices such as cameras and sensors.

[0539] "Reflecting in a virtual space" refers to the procedure of displaying or embodiing user actions and choices in a virtual reality environment in real time.

[0540] "Analyzing speech" refers to the process of using speech recognition technology to capture user speech data and extract its characteristics and meaning.

[0541] "Intelligent processing means" refers to a system or method for performing certain data processing, analysis, and evaluation using artificial intelligence.

[0542] "Recognizing emotional states" refers to methods of detecting or estimating a user's psychological or emotional state through their voice and actions.

[0543] "Adjusting feedback" refers to dynamically changing the content of the information and advice provided based on the user's situation and evaluation data.

[0544] "Recording generated information" refers to the process of saving digital data created or implemented by the user so that it can be used or shared later.

[0545] "Receiving feedback from other users" refers to the process of receiving ratings and comments from different users via the cloud or network.

[0546] This invention is a system for users to obtain an interactive experience in a virtual reality space. The embodiments thereof are described below.

[0547] First, the server is responsible for generating the virtual reality environment. Possible platforms for this include game engines such as Unity and Unreal Engine. These engines are used to build themes and stage designs in real time based on user selections. The server also incorporates intelligent processing capabilities to receive and analyze audio and motion data transmitted from the user. For audio analysis, it uses an audio processing library such as Librosa to extract audio characteristics. Furthermore, it uses an emotion analysis API (e.g., a natural language processing service) to evaluate the user's emotional state.

[0548] Next, the terminal plays a role in collecting the user's movements and voice. This is achieved using a VR headset or motion capture device (e.g., Kinect, Leap Motion). The data acquired from the terminal is reflected in the user's avatar in the virtual space, enabling dynamic interaction. The terminal also provides the user with visual and auditory feedback based on evaluation results sent from the server. This allows the user to obtain information that helps improve their performance in real time.

[0549] Finally, users can adjust and improve their performance based on real-time feedback. For example, if a user sings emotionally in a virtual concert, the system can detect that emotion and provide precise expression guidance and emotional advice. Users can also record their performances and share them with other users to receive more diverse feedback and deepen communication.

[0550] For example, when a user sings an upbeat pop song, the excitement in their voice and the energy of their movements are analyzed. If the user appears to be enjoying themselves, feedback is provided on how to further enhance that enjoyment, and the virtual audience's reaction is also designed to be positive.

[0551] As an example of a prompt, the generative AI model can be used with the following prompt: "Explain how to analyze a user's emotions and provide feedback when they sing an upbeat pop song in a virtual reality space." This allows the entire system to significantly improve the user's immersive experience.

[0552] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0553] Step 1:

[0554] The user puts on a VR headset and motion capture device and begins the virtual reality experience. The user's account information and initial settings are used as input. The connection to the virtual reality environment is configured as output.

[0555] Step 2:

[0556] The server generates a virtual reality environment based on user selections. The input includes settings such as the user's chosen stage theme and design. The output is a virtual space constructed using Unity or Unreal Engine.

[0557] Step 3:

[0558] The terminal captures the user's movements using a motion capture device. The user's physical movement data is acquired as input. The captured movement data is sent to the server as output and reflected in the avatar within the virtual space.

[0559] Step 4:

[0560] The device collects the user's voice through the microphone. The user's voice data is acquired as input. The collected voice data is sent to the server as output.

[0561] Step 5:

[0562] The server analyzes the received audio data. As input, the audio data is analyzed using an audio processing library such as Librosa. Data processing extracts characteristics such as pitch, rhythm, and volume. The analyzed audio characteristics data is obtained as output.

[0563] Step 6:

[0564] The server uses an emotion analysis API to evaluate the user's emotional state. Voice characteristic data and behavioral data are used as input. Emotion is estimated from voice tone and behavioral dynamics as part of the data calculation. Emotional evaluation data is generated as output.

[0565] Step 7:

[0566] The server generates feedback based on sentiment evaluation data. Speech characteristic data and sentiment evaluation data are used as input. Visual or auditory feedback information is created as output and sent to the terminal.

[0567] Step 8:

[0568] The terminal displays feedback received from the server to the user. Feedback information is acquired as input. UI changes and voice advice within the virtual space are displayed as output.

[0569] Step 9:

[0570] The user adjusts their performance based on the feedback provided. Feedback information is used as input. The output is an improvement in the user's performance, both emotionally and technically.

[0571] Step 10:

[0572] Users can record their performance and share it with other users. Audio and motion data from the session are used as input. As output, the generated content is saved to the cloud and can be shared with other participants.

[0573] (Application Example 2)

[0574] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0575] Traditional virtual reality technologies have made it difficult for users to receive emotionally stimulating feedback in real time to improve their performance. Furthermore, they have been unable to replicate realistic reactions from other users in a virtual space, resulting in a lack of immersion and interactivity. This has limited opportunities for users to proactively interact and enhance their self-expression.

[0576] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0577] In this invention, the server includes means for generating a virtual reality space, means for capturing user actions and reflecting them in the virtual reality space, and knowledge processing means for recognizing and evaluating the user's voice in real time. This allows the user to receive feedback that takes into account not only technical evaluations but also emotional aspects, realizing a highly immersive and interactive experience.

[0578] A "virtual reality space" is a three-dimensional virtual environment created using digital technology, an artificial space for users to have an immersive experience.

[0579] "Means for capturing user actions" refer to technologies and devices that detect the user's body movements in real time and analyze them as digital data.

[0580] A "knowledge processing tool" is a process that uses artificial intelligence technology to analyze user voice and actions and provide evaluations and feedback.

[0581] The "emotional aspect" refers to elements related to the user's emotions and mental state, and is the aspect that influences feedback and interaction.

[0582] "Providing appearances or stage designs according to user selection" means automatically setting visual themes and stage configurations in a virtual reality space based on the user's preferences and tastes.

[0583] "Virtually recreating other users' reactions" means providing an interactive environment by visually or audibly representing reactions that other users have never actually made in reality.

[0584] The system for implementing this invention provides a user performance experience using a virtual reality space. A server generates a virtual reality space and captures the user's actions and voice in real time within this space. This uses a camera and microphone installed on the user's device. The captured action and voice data are transferred to the server and analyzed using artificial intelligence technology, which is a knowledge processing tool. Software used in this system includes Google Cloud Speech-to-Text API and Microsoft Azure Text Analytics API.

[0585] The server sets up visual or auditory feedback based on the analysis results and sends it to the user in real time. An emotion engine is also used to determine the user's emotional state, and the feedback is adjusted from an emotional perspective. In this process, the setting or stage design is provided according to the user's choices, and immersion is enhanced by virtually recreating the reactions of other users.

[0586] As a concrete example, when a user performs a jazz piece, they can receive positive feedback from other users acting as a virtual audience. Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and movements during the performance and suggests more effective ways to express themselves. An example of a prompt might be, "By performing an upbeat jazz piece with rich emotion, what kind of emotions can you convey to the audience? Please provide some feedback examples."

[0587] The device provides users with visual and auditory feedback based on data received from the server. This allows users to attempt self-expression and see the results in real time. The invention enables users to improve their technical and emotional performance and enjoy interactive communication with other users.

[0588] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0589] Step 1:

[0590] The user logs into the virtual reality space using a terminal. Inputs include a user ID and password, and a selection of a virtual stage is also made. The output is initial setup information based on the selected stage. At this stage, the user reviews various settings and prepares for performance.

[0591] Step 2:

[0592] The device uses a camera and microphone to capture the user's actions and voice in real time. Input includes user movement data and voice data, which are sent to the server. Output is digitized movement data and voice data. User movements are captured by the camera, and voice is recorded by the microphone.

[0593] Step 3:

[0594] The server analyzes the transmitted data using knowledge processing tools. Inputs include behavioral and audio data, from which pitch, rhythm, and emotional information are analyzed. Outputs include evaluation results and emotional feedback. The data is analyzed using the Google Cloud Speech-to-Text API for speech analysis and the Microsoft Azure Text Analytics API for emotional analysis.

[0595] Step 4:

[0596] The server generates and sends feedback to the terminal based on the analysis results. The inputs used are the analyzed evaluation results and sentiment information. The output is feedback data delivered visually and aurally. The feedback is visualized as UI changes and voice advice, and actions are taken to enhance the user experience.

[0597] Step 5:

[0598] The device presents feedback received from the server to the user visually and audibly. Feedback data is used as input data. The output consists of messages of improvement and praise for the user. In this process, the device uses its display and speakers to deliver information and support the user's actions to learn from and enjoy the experience.

[0599] Step 6:

[0600] Based on the feedback received through their performance, users make adjustments and preparations for their next performance. The input is the content of the feedback, and the output is the performance plan that the user uses to make adjustments. Users analyze the received information and set goals for the next stage.

[0601] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0602] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0603] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0604] [Fourth Embodiment]

[0605] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0606] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0607] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0608] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0609] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0610] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0611] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0612] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0613] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0614] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0615] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0616] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0617] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0618] The system of the present invention is configured to enable users to experience an immersive concert in a virtual reality space. This system functions through the cooperation of a server, a terminal, and the user. The following describes each element and specific embodiments.

[0619] The server is responsible for generating the virtual reality environment and constructing themes and stage designs according to the user's selections. When a user chooses a specific stage design, the server prepares 3D models and effect data corresponding to that design and streams them to the terminal. At this time, the server receives the user's voice data and performs real-time voice analysis using artificial intelligence to evaluate the singing.

[0620] The terminal allows users to access the system using a virtual reality device and interact with it in a natural way. The terminal has the capability to capture user movements and reflect them in the virtual space. It also receives feedback based on user performance evaluations sent from the server and presents it to the user in an appropriate format. For example, if a user sings off-key, that part is visually highlighted.

[0621] After logging into the system, users can realize their performance in a virtual reality space while making various settings. Users can select songs from the content library provided by the system and start singing. They can also use their devices to receive real-time feedback, adjust their singing, and strive to improve their skills.

[0622] As a concrete example, suppose the user selects a rock concert stage. In this case, the server generates a rock-themed stage set, and the device reflects the user's movements in real time. When the user sings a song, artificial intelligence analyzes the voice in real time and provides accurate feedback. Based on this feedback, the user improves their singing.

[0623] Furthermore, this system also includes a feature that allows users to share the performance they generate with other users. The server stores user performance data and makes it available for sharing within the community. This expands opportunities for users to express themselves through feedback and evaluations from other users.

[0624] The following describes the processing flow.

[0625] Step 1:

[0626] The user logs into the system. The terminal obtains the user's authentication information from the input screen and sends it to the server. The server uses the received authentication information to refer to the database and authenticate the user's account.

[0627] Step 2:

[0628] The user selects a theme and stage for the virtual concert. Based on the user's selection, the server prepares the corresponding 3D models and effect data and gets ready to transfer them to the terminal.

[0629] Step 3:

[0630] The device initializes the VR device and tracks the user's position and movements in real time. This ensures that the user's movements are accurately reflected in the virtual reality space.

[0631] Step 4:

[0632] The user selects a song to sing. The device requests the selected song data from the server and stores the data in the cache as needed.

[0633] Step 5:

[0634] When the user starts singing, the device captures the user's voice from the microphone and streams that audio data to the server in real time.

[0635] Step 6:

[0636] The server uses artificial intelligence to analyze the received audio data and evaluates the user's singing based on various evaluation criteria such as pitch, rhythm, and volume.

[0637] Step 7:

[0638] The server generates evaluation results and sends them to the terminal as feedback. The terminal then presents this feedback to the user visually or audibly.

[0639] Step 8:

[0640] The user adjusts their singing based on the feedback and practices further. If necessary, the device records the user's performance and saves it for later playback.

[0641] Step 9:

[0642] If a user chooses to share their performance with the community, the server stores that data and makes it accessible to other users.

[0643] Step 10:

[0644] Other users can view the shared performance and leave comments and ratings. The server notifies the user who originally posted the performance of any newly added comments or ratings.

[0645] (Example 1)

[0646] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0647] Current virtual reality technologies make it difficult for users to experience immersive, real-time performances. Furthermore, there is a lack of immediate feedback to effectively promote individual user skill improvement. Additionally, there are limited means for sharing generated performances with others and stimulating communication.

[0648] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0649] In this invention, the server includes means having the function of generating a virtual reality space, means having the function of recording the user's activities and reflecting them in the virtual environment space, and an artificial intelligence device that recognizes the user's voice and performs analysis and evaluation in real time. As a result, the user can enjoy an immersive performance experience in real time, improve their skills by receiving immediate feedback, and effectively share the generated performance with other users.

[0650] A "virtual reality space" is a three-dimensional environment generated by a computer, in which users can have an immersive experience in real time.

[0651] An "information processing device" is an electronic device that performs data calculations, analysis, streaming, and other processing to enable virtual reality experiences.

[0652] "User activity" refers to the collective actions and voice inputs performed by the user within the virtual reality space, which are reflected in the virtual environment through the interface.

[0653] An "artificial intelligence device" is a set of software or hardware that analyzes user input such as voice and movement in real time through data analysis and learning, and provides evaluation and feedback.

[0654] "Subject" refers to a theme or concept in a virtual reality space, and is an element that expresses different styles and atmospheres that can be selected by the user.

[0655] "Spatial design" refers to the arrangement and design of environments and stages within a virtual reality space, and is an element that should be customized according to the user's preferences.

[0656] This invention is a system that enables users to conduct immersive performance experiences in a virtual reality space. Specific embodiments of each element are shown below.

[0657] The server plays a central role in generating the virtual reality space. Based on the theme selected by the user, the server prepares 3D designs and effects. For example, if the user chooses a rock concert theme, the server generates data for a rock-style stage set and streams it to the terminal. The server uses a generative AI model, particularly when analyzing audio data in real time. This generative AI model analyzes the user's voice and provides evaluations based on pitch and rhythm.

[0658] The terminal provides an interface for users to access the system through a virtual reality device. This terminal has the function of capturing user actions and reflecting them in the virtual reality space. It also visually displays feedback based on voice analysis results, informing the user in real time. A distinctive feature of this system is that the terminal immediately reflects feedback, assisting in improving user performance.

[0659] After logging into the system, users can select a song from the provided music library and begin performing in a virtual reality space. Users receive feedback from their device and adjust their singing in real time, allowing them to improve their skills.

[0660] A concrete example would be a prompt message such as, "Provide a 3D model and effects based on the stage design selected by the user, and perform real-time voice analysis." In this way, each element works together to provide an intuitive and effective experience for the user.

[0661] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0662] Step 1:

[0663] The server receives the user's login information and generates a virtual reality space based on the selected theme. Once the login information and theme selection are entered, the server prepares 3D model data and effect data. For example, if the user selects "Rock Concert," the server generates a rock-themed stage design and prepares the necessary data.

[0664] Step 2:

[0665] The server streams the generated virtual reality data to the terminal. Here, 3D model data and effect data are input, and compressed data is output to the terminal. Specifically, a streaming protocol is used to efficiently deliver the data, allowing users to experience it without delay.

[0666] Step 3:

[0667] The device displays a virtual reality space to the user based on data received from the server. Compressed data from the server is the input, and the 3D space displayed on the virtual reality device is the output. Specifically, the avatar's movement on the stage is rendered in real time in response to the user's movements.

[0668] Step 4:

[0669] The device captures the user's movements and voice and sends them to the server. Here, user movement information and voice data are inputs, and raw data is output to the server. Specifically, data is collected using motion-tracking sensors and a microphone.

[0670] Step 5:

[0671] The server uses a generative AI model to analyze speech in real time and generate evaluation results. User speech data is the input, and the analyzed evaluation data is the output. Specifically, detailed analysis of pitch and rhythm is performed, and feedback information is generated.

[0672] Step 6:

[0673] The terminal receives feedback from the server and presents it to the user visually or audibly. The analyzed evaluation data is the input and becomes the visual or audible output to the user. Specifically, it displays off-key parts graphically and provides voice guidance indicating areas for improvement.

[0674] (Application Example 1)

[0675] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0676] The present invention aims to enable individual users to easily experience immersive virtual reality regardless of location, provide opportunities for improvement through real-time voice analysis and motion feedback, and create an environment that facilitates the sharing of experiences and exchange of feedback among users. In particular, it aims to enable a sufficiently immersive experience even in a home environment and improve access flexibility by utilizing smartphones.

[0677] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0678] In this invention, the server includes means for generating a virtual reality environment, means for detecting user actions and reflecting them in the virtual reality space, and intelligent processing means for analyzing and evaluating the user's voice in real time. This enables the provision of a high-quality immersive experience through a home environment or smartphone, and by utilizing voice analysis and feedback functions, it is possible to improve user performance and expand the experience.

[0679] A "virtual reality environment" is a digital world created by a computer in a 3D space that users can immerse themselves in and experience.

[0680] "Means for detecting user actions" refers to technologies that sense the movements of a user's body or device in real time and reflect them in the digital space.

[0681] "Intelligent processing means" refers to information processing technology that has the ability to analyze data such as user voice and make judgments and evaluations.

[0682] An "immersive experience" refers to an experience in which the user forgets physical reality and enters a virtual reality world, utilizing all senses such as sight and hearing.

[0683] "Feedback" is the process of providing users with evaluations and responses to their statements and actions, and it plays a role in encouraging improvement and new discoveries.

[0684] "Home environment" refers to the living space within a typical house, a place where daily use is expected.

[0685] A "smartphone" is a portable communication device that has communication capabilities and operates as a multi-functional computer.

[0686] To implement this invention, a system is built through the cooperation of a server, a terminal such as a smartphone, and a user. The server prepares the 3D spatial data and theme design necessary to generate the virtual reality environment and streams it to the terminal. To detect user movements, the terminal uses sensors from a smartphone or VR device to capture the user's movements in real time and reflect them in the virtual environment. In this process, 3D rendering is performed using software such as Unity or Unreal Engine.

[0687] Furthermore, the system utilizes speech recognition services such as Google Cloud Speech-to-Text as an intelligent processing tool to analyze the user's speech in real time. The analysis results are provided to the user as visual or auditory feedback, encouraging improvements in pitch and rhythm. This entire process creates an immersive concert experience.

[0688] As a concrete example, when a user selects an "Anime Song Festival" theme and sings a song, a 3D stage provided by the server is displayed on the terminal. The user's voice data is instantly processed intelligently, and feedback based on pitch and volume appears on the screen along with visual effects. In this way, the user can improve their performance within the virtual environment.

[0689] An example of a prompt from the generating AI is, "Please tell me any particular vocal points or performance aspects you should pay attention to on the stage of the anime song festival you have chosen." Through this prompt, the user can receive more precise feedback and advice from the generating AI.

[0690] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0691] Step 1:

[0692] The user logs into their device and selects a theme for the virtual reality concert. The user's theme selection information is passed to the device as input and sent as a request to the server. The server generates the necessary 3D environment data and stage design based on the selected theme and streams it to the device. As output, the 3D model and effect data are presented to the user's device.

[0693] Step 2:

[0694] The terminal captures the user's movements in real time using the sensors of the VR device or smartphone worn by the user. The user's movement information is input to the terminal, and data processing is performed using Unity or Unreal Engine technologies to reflect the movements in real time within the virtual reality space. The reflected user movements are then displayed in 3D space as output.

[0695] Step 3:

[0696] The user sings a song of their choice, and the device collects the audio data through its microphone. The user's audio data is captured by the device as input and sent to a server. The server uses a speech recognition service such as Google Cloud Speech-to-Text to analyze the audio data in real time. The analysis results are generated as feedback data as output.

[0697] Step 4:

[0698] The server generates feedback on pitch and rhythm based on the analysis results and sends it to the terminal. The input is the feedback information generated by the analysis, provided from the server to the terminal. The feedback is presented visually as highlighting and sound effects, and is used as educational material for the user. The output is information about areas for improvement in their own performance.

[0699] Step 5:

[0700] The user improves their singing based on feedback and performs again. The improved singing, based on the feedback, is processed again by the system and saved for sharing and evaluation with other users. As input, the improved performance data is entered into the terminal. As output, the user's improved performance is saved and shared, and feedback from other users can be received.

[0701] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0702] The system of this invention tracks the user's movements and voice as they engage in a real-time concert experience in a virtual reality space, and provides feedback based on this data. Furthermore, by incorporating an emotion engine, the system recognizes the user's emotions and adjusts the feedback from an emotional perspective.

[0703] The server generates a virtual reality space and prepares stage designs and themes according to the user's selection. This allows users to express themselves in diverse stage settings. The server also receives the user's voice and movement data in real time and evaluates it using artificial intelligence and an emotion engine. Voice data is analyzed based on criteria such as pitch, rhythm, and volume, and the emotion engine determines the user's emotional state from the characteristics of the voice and movements.

[0704] The device captures the user's physical movements and reflects them in the virtual space. When the user performs a specific movement, that movement is reflected in real time on the user avatar on the stage, enabling interactive performances. The device also receives evaluations and emotional feedback sent from the server and presents it to the user. The feedback is displayed as visual UI changes and audio advice.

[0705] Users can receive real-time emotional and technical feedback through their performance. For example, if a user sings a ballad with great emotion, the system will detect that emotion and provide more detailed expression guidance and emotional advice. This allows users to improve not only their technical skills but also their emotional expression. Furthermore, users can record their performances and share them with others, broadening the scope of feedback and deepening their interaction with the community.

[0706] For example, when a user sings an upbeat pop song, the emotion engine analyzes the excitement in the user's voice and the energy of their movements. If the user appears to be enjoying themselves, the feedback provides know-how to further enhance that enjoyment, and the virtual audience's reaction also becomes positive. In this way, the system provides comprehensive support that takes the user's emotions into account.

[0707] The following describes the processing flow.

[0708] Step 1:

[0709] The user logs into the system and puts on a virtual reality device. The device collects the user's authentication information and sends it to the server. The server authenticates the user's account based on this information and starts a session.

[0710] Step 2:

[0711] The user selects the stage design and theme for the virtual concert. The server prepares the corresponding 3D models and effects based on the user's selection and streams them to the device.

[0712] Step 3:

[0713] The device tracks the user's movements through the VR device and reflects them in the virtual reality space in real time. This allows the user's movements to be represented on a virtual stage.

[0714] Step 4:

[0715] The user selects a song and begins singing. The device collects the user's voice through the microphone and sends the audio data to the server.

[0716] Step 5:

[0717] The server uses artificial intelligence to analyze the user's voice data, evaluating pitch, rhythm, and volume. Simultaneously, an emotion engine performs sentiment analysis based on the user's voice tone and speed.

[0718] Step 6:

[0719] Based on the analysis results, the server generates technical and emotional feedback. This feedback includes suggestions for improving pitch and enhancing expressiveness.

[0720] Step 7:

[0721] The terminal receives feedback from the server and presents it to the user. Visually, this is done using on-screen instructions and color indicators, while auditorily, it is done using voice messages and guides.

[0722] Step 8:

[0723] Users continue singing while viewing and listening to feedback. Based on emotional feedback, users adjust their expression, aiming for a more emotionally rich performance.

[0724] Step 9:

[0725] After a singing session ends, users can save their performance data and share it with the community. The server accepts this data and makes it public so other users can access it.

[0726] Step 10:

[0727] Other users can watch shared performances and post reactions and comments. The server notifies the original user of these new interactions to encourage further feedback.

[0728] (Example 2)

[0729] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0730] Traditional virtual reality systems have limitations in tracking and evaluating user actions and voice in real time, making it difficult to accurately capture emotions and thus hindering improvements in the user experience. Furthermore, there are challenges in effectively sharing generated content and obtaining sufficient feedback from other users.

[0731] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0732] In this invention, the server includes means for generating a virtual reality environment, means for tracking the user's actions and reflecting them in the virtual space, and intelligent processing means for analyzing the user's voice and evaluating it in real time. This enables the provision of feedback based on advanced emotion recognition in real time and the efficient sharing of generated content.

[0733] A "virtual reality environment" is a three-dimensional virtual space created by a computer, an environment in which users can have an immersive experience.

[0734] "Tracking user movements" refers to the process of detecting and recording a user's physical movements using devices such as cameras and sensors.

[0735] "Reflecting in a virtual space" refers to the procedure of displaying or embodiing user actions and choices in a virtual reality environment in real time.

[0736] "Analyzing speech" refers to the process of using speech recognition technology to capture user speech data and extract its characteristics and meaning.

[0737] "Intelligent processing means" refers to a system or method for performing certain data processing, analysis, and evaluation using artificial intelligence.

[0738] "Recognizing emotional states" refers to methods of detecting or estimating a user's psychological or emotional state through their voice and actions.

[0739] "Adjusting feedback" refers to dynamically changing the content of the information and advice provided based on the user's situation and evaluation data.

[0740] "Recording generated information" refers to the process of saving digital data created or implemented by the user so that it can be used or shared later.

[0741] "Receiving feedback from other users" refers to the process of receiving ratings and comments from different users via the cloud or network.

[0742] This invention is a system for users to obtain an interactive experience in a virtual reality space. The embodiments thereof are described below.

[0743] First, the server is responsible for generating the virtual reality environment. Possible platforms for this include game engines such as Unity and Unreal Engine. These engines are used to build themes and stage designs in real time based on user selections. The server also incorporates intelligent processing capabilities to receive and analyze audio and motion data transmitted from the user. For audio analysis, it uses an audio processing library such as Librosa to extract audio characteristics. Furthermore, it uses an emotion analysis API (e.g., a natural language processing service) to evaluate the user's emotional state.

[0744] Next, the terminal plays a role in collecting the user's movements and voice. This is achieved using a VR headset or motion capture device (e.g., Kinect, Leap Motion). The data acquired from the terminal is reflected in the user's avatar in the virtual space, enabling dynamic interaction. The terminal also provides the user with visual and auditory feedback based on evaluation results sent from the server. This allows the user to obtain information that helps improve their performance in real time.

[0745] Finally, users can adjust and improve their performance based on real-time feedback. For example, if a user sings emotionally in a virtual concert, the system can detect that emotion and provide precise expression guidance and emotional advice. Users can also record their performances and share them with other users to receive more diverse feedback and deepen communication.

[0746] For example, when a user sings an upbeat pop song, the excitement in their voice and the energy of their movements are analyzed. If the user appears to be enjoying themselves, feedback is provided on how to further enhance that enjoyment, and the virtual audience's reaction is also designed to be positive.

[0747] As an example of a prompt, the generative AI model can be used with the following prompt: "Explain how to analyze a user's emotions and provide feedback when they sing an upbeat pop song in a virtual reality space." This allows the entire system to significantly improve the user's immersive experience.

[0748] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0749] Step 1:

[0750] The user puts on a VR headset and motion capture device and begins the virtual reality experience. The user's account information and initial settings are used as input. The connection to the virtual reality environment is configured as output.

[0751] Step 2:

[0752] The server generates a virtual reality environment based on user selections. The input includes settings such as the user's chosen stage theme and design. The output is a virtual space constructed using Unity or Unreal Engine.

[0753] Step 3:

[0754] The terminal captures the user's movements using a motion capture device. The user's physical movement data is acquired as input. The captured movement data is sent to the server as output and reflected in the avatar within the virtual space.

[0755] Step 4:

[0756] The device collects the user's voice through the microphone. The user's voice data is acquired as input. The collected voice data is sent to the server as output.

[0757] Step 5:

[0758] The server analyzes the received audio data. As input, the audio data is analyzed using an audio processing library such as Librosa. Data processing extracts characteristics such as pitch, rhythm, and volume. The analyzed audio characteristics data is obtained as output.

[0759] Step 6:

[0760] The server uses an emotion analysis API to evaluate the user's emotional state. Voice characteristic data and behavioral data are used as input. Emotion is estimated from voice tone and behavioral dynamics as part of the data calculation. Emotional evaluation data is generated as output.

[0761] Step 7:

[0762] The server generates feedback based on sentiment evaluation data. Speech characteristic data and sentiment evaluation data are used as input. Visual or auditory feedback information is created as output and sent to the terminal.

[0763] Step 8:

[0764] The terminal displays feedback received from the server to the user. Feedback information is acquired as input. UI changes and voice advice within the virtual space are displayed as output.

[0765] Step 9:

[0766] The user adjusts their performance based on the feedback provided. Feedback information is used as input. The output is an improvement in the user's performance, both emotionally and technically.

[0767] Step 10:

[0768] Users can record their performance and share it with other users. Audio and motion data from the session are used as input. As output, the generated content is saved to the cloud and can be shared with other participants.

[0769] (Application Example 2)

[0770] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0771] Traditional virtual reality technologies have made it difficult for users to receive emotionally stimulating feedback in real time to improve their performance. Furthermore, they have been unable to replicate realistic reactions from other users in a virtual space, resulting in a lack of immersion and interactivity. This has limited opportunities for users to proactively interact and enhance their self-expression.

[0772] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0773] In this invention, the server includes means for generating a virtual reality space, means for capturing user actions and reflecting them in the virtual reality space, and knowledge processing means for recognizing and evaluating the user's voice in real time. This allows the user to receive feedback that takes into account not only technical evaluations but also emotional aspects, realizing a highly immersive and interactive experience.

[0774] A "virtual reality space" is a three-dimensional virtual environment created using digital technology, an artificial space for users to have an immersive experience.

[0775] "Means for capturing user actions" refer to technologies and devices that detect the user's body movements in real time and analyze them as digital data.

[0776] A "knowledge processing tool" is a process that uses artificial intelligence technology to analyze user voice and actions and provide evaluations and feedback.

[0777] The "emotional aspect" refers to elements related to the user's emotions and mental state, and is the aspect that influences feedback and interaction.

[0778] "Providing appearances or stage designs according to user selection" means automatically setting visual themes and stage configurations in a virtual reality space based on the user's preferences and tastes.

[0779] "Virtually recreating other users' reactions" means providing an interactive environment by visually or audibly representing reactions that other users have never actually made in reality.

[0780] The system for implementing this invention provides a user performance experience using a virtual reality space. A server generates a virtual reality space and captures the user's actions and voice in real time within this space. This uses a camera and microphone installed on the user's device. The captured action and voice data are transferred to the server and analyzed using artificial intelligence technology, which is a knowledge processing tool. Software used in this system includes Google Cloud Speech-to-Text API and Microsoft Azure Text Analytics API.

[0781] The server sets up visual or auditory feedback based on the analysis results and sends it to the user in real time. An emotion engine is also used to determine the user's emotional state, and the feedback is adjusted from an emotional perspective. In this process, the setting or stage design is provided according to the user's choices, and immersion is enhanced by virtually recreating the reactions of other users.

[0782] As a concrete example, when a user performs a jazz piece, they can receive positive feedback from other users acting as a virtual audience. Furthermore, the emotion engine analyzes the user's emotions from their facial expressions and movements during the performance and suggests more effective ways to express themselves. An example of a prompt might be, "By performing an upbeat jazz piece with rich emotion, what kind of emotions can you convey to the audience? Please provide some feedback examples."

[0783] The device provides users with visual and auditory feedback based on data received from the server. This allows users to attempt self-expression and see the results in real time. The invention enables users to improve their technical and emotional performance and enjoy interactive communication with other users.

[0784] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0785] Step 1:

[0786] The user logs into the virtual reality space using a terminal. Inputs include a user ID and password, and a selection of a virtual stage is also made. The output is initial setup information based on the selected stage. At this stage, the user reviews various settings and prepares for performance.

[0787] Step 2:

[0788] The device uses a camera and microphone to capture the user's actions and voice in real time. Input includes user movement data and voice data, which are sent to the server. Output is digitized movement data and voice data. User movements are captured by the camera, and voice is recorded by the microphone.

[0789] Step 3:

[0790] The server analyzes the transmitted data using knowledge processing tools. Inputs include behavioral and audio data, from which pitch, rhythm, and emotional information are analyzed. Outputs include evaluation results and emotional feedback. The data is analyzed using the Google Cloud Speech-to-Text API for speech analysis and the Microsoft Azure Text Analytics API for emotional analysis.

[0791] Step 4:

[0792] The server generates and sends feedback to the terminal based on the analysis results. The inputs used are the analyzed evaluation results and sentiment information. The output is feedback data delivered visually and aurally. The feedback is visualized as UI changes and voice advice, and actions are taken to enhance the user experience.

[0793] Step 5:

[0794] The device presents feedback received from the server to the user visually and audibly. Feedback data is used as input data. The output consists of messages of improvement and praise for the user. In this process, the device uses its display and speakers to deliver information and support the user's actions to learn from and enjoy the experience.

[0795] Step 6:

[0796] Based on the feedback received through their performance, users make adjustments and preparations for their next performance. The input is the content of the feedback, and the output is the performance plan that the user uses to make adjustments. Users analyze the received information and set goals for the next stage.

[0797] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0798] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0799] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0800] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0801] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0802] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0803] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0804] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0805] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0806] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0807] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0808] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0809] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0810] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0811] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0812] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0813] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0814] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0815] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0816] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0817] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0818] The following is further disclosed regarding the embodiments described above.

[0819] (Claim 1)

[0820] Means for generating a virtual reality environment,

[0821] A means of tracking user actions and reflecting them in a virtual reality space,

[0822] An artificial intelligence method that recognizes and evaluates the user's voice in real time,

[0823] Means for generating visual or auditory feedback based on the aforementioned evaluation,

[0824] A means of sharing user-generated content and receiving feedback from other users,

[0825] A system that includes this.

[0826] (Claim 2)

[0827] The system according to claim 1, wherein the artificial intelligence means is configured to perform evaluations based on pitch, rhythm, and volume.

[0828] (Claim 3)

[0829] The system according to claim 1, wherein the virtual reality space is configured to provide different themes or stage designs according to the user's selection.

[0830] "Example 1"

[0831] (Claim 1)

[0832] An information processing device having the function of generating a virtual reality space,

[0833] An information processing device having the function of recording the user's activities and reflecting them in a virtual environment space,

[0834] An artificial intelligence device that recognizes the user's voice and performs analysis and evaluation in real time,

[0835] An information processing device having the function of generating visual or auditory feedback based on the aforementioned analysis and evaluation,

[0836] An information processing device having the function of sharing information generated by users and receiving evaluations from other users,

[0837] A system that includes this.

[0838] (Claim 2)

[0839] The system according to claim 1, wherein the artificial intelligence device is configured to perform analysis and evaluation based on pitch, rhythm, and volume.

[0840] (Claim 3)

[0841] The system according to claim 1, wherein the virtual environment space is configured to provide different themes or spatial designs according to the user's choice.

[0842] "Application Example 1"

[0843] (Claim 1)

[0844] Means for generating a virtual reality environment,

[0845] A means of detecting user actions and reflecting them in a virtual reality space,

[0846] An intelligent processing means that analyzes and evaluates the user's voice in real time,

[0847] Means for generating a visual or auditory response based on the aforementioned evaluation,

[0848] A means of sharing user-generated information and receiving feedback from other users,

[0849] A means of enabling use in a home environment, which is installed on a smartphone or portable device.

[0850] A system that includes this.

[0851] (Claim 2)

[0852] The system according to claim 1, wherein the intelligent processing means is configured to make decisions based on pitch, rhythm, and volume.

[0853] (Claim 3)

[0854] The system according to claim 1, wherein the virtual reality space is configured to provide different themes or performance designs according to the user's selection.

[0855] "Example 2 of combining an emotion engine"

[0856] (Claim 1)

[0857] Means for generating a virtual reality environment,

[0858] A means of tracking user actions and reflecting them in a virtual space,

[0859] An intelligent processing means that analyzes and evaluates the user's voice in real time,

[0860] Means for generating visual or auditory feedback based on the aforementioned evaluation,

[0861] A means of recognizing the user's emotional state and adjusting feedback based on those emotions,

[0862] A means of recording information generated by users and receiving feedback from other users,

[0863] A system that includes this.

[0864] (Claim 2)

[0865] The system according to claim 1, wherein the intelligent processing means is configured to perform an emotional evaluation based on voice characteristics and body movements.

[0866] (Claim 3)

[0867] The system according to claim 1, wherein the virtual space is configured to provide different themes or stage designs according to the user's choice.

[0868] "Application example 2 when combining with an emotional engine"

[0869] (Claim 1)

[0870] Means for generating a virtual reality space,

[0871] A means of capturing user actions and reflecting them in a virtual reality space,

[0872] A knowledge processing means that recognizes and evaluates the user's voice in real time,

[0873] Means for generating visual or auditory feedback based on the aforementioned evaluation,

[0874] A means for recognizing the user's emotions and adjusting the feedback from an emotional perspective,

[0875] A means of sharing user-generated information and receiving feedback from other users,

[0876] A system that includes this.

[0877] (Claim 2)

[0878] The system according to claim 1, wherein the knowledge processing means is configured to perform evaluations based on pitch, rhythm, and volume.

[0879] (Claim 3)

[0880] The system according to claim 1, wherein the virtual reality space is configured to provide different appearances or stage designs according to the user's selection, and is further configured to virtually reproduce the reactions of other users. [Explanation of symbols]

[0881] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means for generating a virtual reality environment, A means of tracking user actions and reflecting them in a virtual reality space, An artificial intelligence method that recognizes and evaluates the user's voice in real time, Means for generating visual or auditory feedback based on the aforementioned evaluation, A means of sharing user-generated content and receiving feedback from other users, A system that includes this.

2. The system according to claim 1, wherein the artificial intelligence means is configured to perform evaluations based on pitch, rhythm, and volume.

3. The system according to claim 1, wherein the virtual reality space is configured to provide different themes or stage designs according to the user's selection.

Images