system

Abstract

A system is provided.SOLUTION: A system, comprising: means for transmitting operation information acquired from an AR terminal to a server to implement interaction between different dimensions; means for analyzing the operation information and inferring an action; means for generating a feedback based on the inferred action and transmitting the feedback to a VR terminal; and means for receiving the feedback and providing a tactile or visual response to a VR user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional VR experiences, it is difficult for non-HMD users to communicate with VR users wearing HMDs, and communication is limited to the sharing of visual information. There is a need to overcome this limitation and enable deep sharing of experiences and mutual interaction across different dimensions. Furthermore, there is a lack of methods to effectively communicate the actions of AR users to VR users, making it difficult to achieve a realistic interaction experience. [Means for solving the problem]

[0005] The present invention provides a system for realizing interactions between different dimensions. Specifically, the system includes a means for transmitting operation information acquired from an AR device to a server, a means including a generative AI model that analyzes the operation information and infers actions, a means for generating feedback based on the inferred actions and transmitting it to a VR device, and a means for receiving the feedback and providing a tactile or visual response to the VR user. This system analyzes the AR user's gesture data, classifies specific actions, and provides vibration feedback or a visual response, thereby realizing realistic interactions between different dimensions.

[0006] An "AR terminal" is a device that uses augmented reality technology to display digital information overlaid on the real world.

[0007] "Operation information" is data that indicates specific gestures and actions performed by the AR user, and includes location information and action history.

[0008] A "server" is a computer system that communicates with AR and VR terminals via a network and receives, analyzes, and transmits data.

[0009] A "generative AI model" is an artificial intelligence model that uses machine learning algorithms to analyze input operation information and infer user actions.

[0010] "Actions" refer to gestures or operations that the AR user performs on the VR user, including specific actions such as "stroking" or "hitting."

[0011] "Feedback" refers to the response provided to the VR user based on the AR user's actions, including haptic feedback and visual responses.

[0012] A "VR terminal" is a device that uses virtual reality technology to provide users with an immersive virtual space, and mainly includes a head-mounted display (HMD) and a vibration device.

[0013] "Haptic feedback" provides a physical sensation to VR users, primarily through device vibrations, providing feedback that allows users to experience certain actions.

[0014] "Visual response" refers to feedback that is visually presented to the user within the VR space, and expresses actions through animations and graphics. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention provides a system for realizing interactions between different dimensions. This system consists of an AR terminal, a server, and a VR terminal, and its main processing steps include capturing operation information, transmitting data, analyzing data, generating feedback, and applying feedback.

[0037] System configuration

[0038] AR device

[0039] AR devices are devices that use augmented reality technology to display digital information overlaid on the real world. These devices are equipped with cameras and sensors that capture the position and movement history of the AR user's hands and fingers in real time.

[0040] server

[0041] The server is a high-performance computer system that communicates with the AR and VR devices via a network. The server is equipped with a generative AI model that analyzes the operation information received from the AR device and infers the user's actions.

[0042] VR device

[0043] A VR device is a device that uses virtual reality technology to provide an immersive virtual space. Its main components include a head-mounted display (HMD) and a vibration device. The VR device provides tactile or visual responses to the VR user based on feedback signals sent from the server.

[0044] Program processing and examples

[0045] AR device operation capture

[0046] The AR device captures gestures made by the AR user toward the VR user's character. For example, if the AR user uses a tablet to make a gesture of patting the VR user's character's head through the screen, the sensor will capture the hand movement and position information.

[0047] Sending and Receiving Data

[0048] The AR device transmits the captured operation information to the server, which receives the operation information and stores it in a data queue.

[0049] Data analysis and behavioral inference

[0050] The server uses the generative AI model to analyze the received operation information. For example, it can infer the action of "stroking" based on the hand's trajectory, speed, and movement pattern. The generative AI model then classifies the action using gesture classes it has learned in advance.

[0051] Generate and send feedback

[0052] The server generates feedback based on the analysis results. For example, if it detects a "stroking" action, it generates an appropriate vibration pattern or haptic feedback and sends that signal to the VR device.

[0053] Applying Feedback

[0054] The VR device provides a response to the VR user based on the feedback signal received from the server. Specifically, the VR user's head-mounted device (HMD) generates a slight vibration to recreate the feeling of being stroked. Additionally, an animation of the user being stroked is displayed in the VR space as visual feedback.

[0055] Specific examples

[0056] For example, suppose an AR user uses a tablet to make a gesture of stroking the VR user's head. The AR device captures this gesture and sends it to the server. The server's generative AI model analyzes this data and infers the action of "stroking." The server then generates an appropriate vibration pattern and sends it to the VR device. The VR device receives this feedback and generates a slight vibration in the VR user's HMD while simultaneously displaying an animation of the stroking. This allows the VR user to experience the sensation of actually being stroked.

[0057] This system allows AR and VR users to communicate intuitively and effectively, even though they are in different dimensions.

[0058] The processing flow will be explained below.

[0059] Step 1:

[0060] The AR device captures the gestures made by the AR user. Specifically, the tablet's camera and sensors capture the AR user's hand position (x, y, z coordinates) and movement (velocity, acceleration) in real time. This information is captured frame by frame.

[0061] Step 2:

[0062] The AR device sends the captured operation information to the server, which uses a real-time communication protocol (e.g., WebSocket) to transmit data with low latency.

[0063] Step 3:

[0064] The server receives the operation information sent from the AR device. The received data is temporarily stored in a queue and prepared for analysis by the generative AI model.

[0065] Step 4:

[0066] The server uses the generative AI model to analyze the received operation information. Specifically, it infers the AR user's actions from hand movements and position information. In this process, it uses pre-trained gesture classes to classify actions such as "stroking" and "hitting."

[0067] Step 5:

[0068] The server generates feedback signals based on the analysis results. For example, if the action of "stroking" is detected, it generates appropriate vibration patterns or visual feedback to reproduce the sensation of stroking.

[0069] Step 6:

[0070] The server transmits the generated feedback signals to the VR device, again using a real-time communication protocol to transmit the signals with low latency.

[0071] Step 7:

[0072] The VR device receives the feedback signal sent from the server, which is used as an instruction to provide tactile and visual responses to the VR user.

[0073] Step 8:

[0074] The VR device applies feedback by vibrating the VR user's head-mounted device (HMD) and also provides visual feedback by displaying an animation of the object being stroked in the VR space.

[0075] Step 9:

[0076] The VR user feels the vibrations in their head and visual feedback, allowing them to feel as if the AR user is stroking them, creating a mutual interaction experience that transcends the dimensions of AR and VR.

[0077] Example 1

[0078] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0079] Conventional virtual reality (VR) and augmented reality (AR) systems have the problem of not intuitively reflecting interactions between users in real time. In particular, it is difficult to provide appropriate tactile and visual feedback when multiple users in different dimensions participate in simultaneous communication. Furthermore, accurate recognition of gestures and the quality of feedback can be reduced, resulting in a poor user experience.

[0080] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0081] In this invention, the server includes a means for transmitting operation information acquired from the AR terminal to the server, a means including a generative AI model for analyzing the operation information and inferring an action, and a means for generating feedback based on the inferred action and transmitting the feedback to the VR terminal, thereby enabling intuitive and effective interaction even between users in different dimensions.

[0082] An "AR terminal" is a device that uses augmented reality technology to overlay digital information onto the real world.

[0083] A "server" is a high-performance computer system that communicates with AR and VR terminals via a network.

[0084] "Operation information" is data related to the user's movements and gestures, and is information captured in real time by cameras and sensors.

[0085] A "generative AI model" is a machine learning model that analyzes received operation information and infers user actions, and uses pre-trained gesture classes.

[0086] "Feedback" refers to reaction data generated based on the analysis results, and includes tactile and visual reactions.

[0087] A "VR terminal" is a device that uses virtual reality technology to provide an immersive virtual space, and includes a head-mounted display and a vibration device.

[0088] "Tactile feedback" refers to feedback that provides a direct sensation to the user's body, and includes physical stimuli such as vibrations.

[0089] "Visual response" refers to providing direct feedback to the user's vision, such as on-screen animation.

[0090] A "gesture" is an operation based on the movement of a user's hands or fingers, and is a motion pattern that indicates a specific action.

[0091] A "network" is an infrastructure for realizing communication between a server and a terminal, and includes the Internet and a local area network (LAN).

[0092] A "head-mounted display (HMD)" is a display device worn by the user on the head, providing an immersive VR experience.

[0093] This invention is a system for realizing interactions between different dimensions, and is mainly composed of an AR terminal, a server, and a VR terminal. Each component and its function will be explained in detail below.

[0094] AR device

[0095] AR devices are devices that use augmented reality technology to overlay digital information onto the real world. Specifically, they are equipped with cameras and sensors and have the ability to capture the positional information and movement history of the user's hands and fingers in real time. For example, an AR-enabled tablet can be used, and it can accurately capture the gestures the user makes with a virtual character.

[0096] server

[0097] The server is a high-performance computer system that communicates with the AR and VR devices via a network. It contains a generative AI model that analyzes the operation information sent from the AR device and infers the user's actions. The generative AI model classifies the actions using pre-trained gesture classes and sends the results to the feedback generation module.

[0098] VR device

[0099] A VR device is a device that uses virtual reality technology to provide an immersive virtual space. Its main components are a head-mounted display (HMD) and a vibration device, and it has the function of providing tactile and visual responses to the VR user based on feedback signals sent from the server. Specifically, the VR user's HMD vibrates to reproduce the feeling of being stroked, and displays a stroke animation as visual feedback.

[0100] Specific examples

[0101] For example, if an AR user uses a tablet to make a gesture of stroking the head of a VR user through the screen, the process proceeds as follows: The user operates the AR device, and their hand movements are captured by the camera and sensors. The captured operation information is sent to the server via the network, and the server's generative AI model analyzes the data and infers the action of "stroking." The server then generates an appropriate vibration pattern and visual feedback, which it sends to the VR device. The VR device receives the feedback signal, generates a slight vibration in the HMD, and simultaneously displays an animation of the user being stroked.

[0102] Prompt Sentence Examples

[0103] Specifically, the following prompts are used:

[0104] "Analyze the following operation information to generate feedback as if the VR user's character is being stroked: hand movement trajectory, speed, and movement pattern: {hand movement data}"

[0105] This system enables AR and VR users to communicate intuitively and effectively in real time, even though they are in different dimensions.

[0106] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0107] Step 1:

[0108] Operation capture using an AR device

[0109] Input: User gestures

[0110] Output: Hand position information, movement history

[0111] Specific behavior:

[0112] The user operates the AR device and touches a virtual object (e.g., the head of a VR character) through the screen. The camera and sensors on the AR device capture this action in real time. Specifically, the position information and movement history of the hands and fingers are acquired and converted into data.

[0113] Step 2:

[0114] Sending data and receiving it on the server

[0115] Input: Hand position information, movement history

[0116] Output: Sending operation information data, storing received data

[0117] Specific behavior:

[0118] The AR device sends the hand position information and movement history to the server. The data is transferred via the network, and the server receives the data and temporarily stores it in a data queue.

[0119] Step 3:

[0120] Data analysis and behavioral inference

[0121] Input: Operation information data

[0122] Output: Estimated result of the action

[0123] Specific behavior:

[0124] The server uses the generated AI model to analyze the received operation information data. Specifically, it analyzes the trajectory, speed, and movement patterns of the hand movement to infer actions such as "stroking." For example, from an analysis of the smooth movement of the hand and its speed, it can infer with a high degree of probability that the action is "stroking."

[0125] Step 4:

[0126] Generate and send feedback

[0127] Input: Estimated result of the action

[0128] Output: Feedback data

[0129] Specific behavior:

[0130] The server generates appropriate feedback based on the action estimation. For example, a vibration pattern or visual feedback (animation) corresponding to the action of "stroking" is generated. The generated feedback data is sent to the VR device.

[0131] Step 5:

[0132] Applying Feedback

[0133] Input: Feedback data

[0134] Output: Tactile and visual feedback to the user

[0135] Specific behavior:

[0136] The VR device responds to the user based on the feedback data received from the server. The HMD generates appropriate vibrations and displays animations of the character being stroked in the VR space. For example, the visual effect of a character's head being stroked along with a slight vibration gives the user the sensation of actually stroking the character.

[0137] (Application example 1)

[0138] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0139] While the direct interaction between customers and store staff forms part of the value of the shopping experience in a physical store, there are limitations in that not all customers can receive support at the same time. Another issue is that when store staff provide support remotely, the information and experience customers receive is limited. This can lead to customers not being able to fully understand how to use a product or its features.

[0140] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0141] In this invention, the server includes: means for transmitting operation information acquired from the AR device to the server; means including a generative AI model that analyzes the operation information and infers actions; means for generating feedback based on the inferred actions and transmitting it to the VR device; means for receiving the feedback and providing tactile or visual feedback to the VR user; means for a customer to wear AR glasses in a physical store and for a remote store staff member to provide product operation instructions using VR gloves; and means for the server to analyze the customer's behavior, generate appropriate feedback, and display it on the AR glasses. This allows customers to intuitively and effectively understand how to use and use products while providing valuable remote support.

[0142] "Interaction between different dimensions" refers to the interaction and information exchange between users that takes place between a virtual space, which is different from the physical space, and a real space using virtual reality or augmented reality.

[0143] An "AR terminal" is a device that uses augmented reality technology to overlay digital information on the real world, and is equipped with a camera and sensors.

[0144] "Operation information" refers to data acquired by the AR device when it recognizes operations such as user movements and gestures.

[0145] A "server" is a high-performance computer system that communicates with various terminals via a network and analyzes data.

[0146] A "generative AI model" is an algorithm or model that uses artificial intelligence to analyze data and infer user operations and behavior.

[0147] A "VR terminal" is a device that uses virtual reality technology to provide users with an immersive virtual space, and includes head-mounted displays and haptic devices.

[0148] "Feedback" refers to the tactile or visual response that is generated in response to a user's operation or action.

[0149] A "brick and mortar store" is a commercial establishment that exists in a physical location and can be visited in person by customers.

[0150] "AR glasses" are glasses-type devices that use augmented reality technology, allowing users to visually view digital information overlaid on the real world.

[0151] "Remote store staff" refers to store staff who are physically located in a different location from the physical store but who support customers virtually.

[0152] "VR gloves" are glove-shaped devices that allow users to operate the device with their hands in a virtual reality world and provide haptic feedback.

[0153] "Analyzing customer behavior" means collecting data on the operations and actions performed by customers, and analyzing that data to infer specific actions.

[0154] This invention provides a system that uses an AR terminal, a server, and a VR terminal to improve the shopping experience in a physical store. This system allows customers to wear AR glasses in the physical store, and remote store staff can use VR gloves to provide instructions on how to operate products.

[0155] First, a customer wears AR glasses and walks around a physical store. The AR device (e.g., a glasses-type device that uses augmented reality technology) captures customer operation information through cameras and sensors. For example, when a customer looks at a product or makes a gesture of reaching out, that operation information is acquired in real time.

[0156] This operation information is then sent to a server and stored in the server's data queue. The server is a high-performance computer system (e.g., AWS® EC2 instance) that analyzes the received operation information using a generative AI model (e.g., OpenAI®'s GPT-4®). Specifically, the server analyzes hand movements, gaze direction, and movement patterns to infer the action of "selecting a product."

[0157] Based on the analyzed actions, the server generates feedback. A remote store staff member wearing VR gloves (e.g., a glove-type device for operating with hands in a virtual reality world) can use the feedback to demonstrate to the customer how to operate the product and its characteristics. The server generates appropriate feedback (visual and tactile responses) and displays it on the AR glasses, allowing the customer to experience the store staff as if they were right in front of them.

[0158] Here is one concrete example:

[0159] 1. A customer puts on AR glasses in a physical store and looks at a specific product.

[0160] 2. The AR device captures the eye movements and hand gestures and sends the operation information to the server.

[0161] 3. The server's generated AI model analyzes the operation information and infers the action of "selecting a product."

[0162] 4. The server sends notifications to remote store staff based on the analysis results.

[0163] 5. Remote store staff will use VR gloves to virtually demonstrate how to operate the product.

[0164] 6. Feedback is displayed on the customer's AR glasses, allowing them to visually confirm how to use the product and its features.

[0165] Example prompts for generative AI models:

[0166] Analyze the following input data and infer the actions taken by the customer (product selection, inquiry, etc.).

[0167] Input data: { "Action": "Reach out", "Target": "Product code 12345", "Timestamp": "2023-10-01T08:00:00Z"}

[0168] Please output the analysis results in the following format:

[0169] {

[0170] "Action": "Product Selection",

[0171] "Product Code": "12345",

[0172] "Recommended feedback": "View a demonstration of how to use the product"

[0173] }

[0174] The invention makes the in-store shopping experience more intuitive and effective, allowing customers to understand the usage and characteristics of products with real-time support from remote store staff.

[0175] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0176] Step 1:

[0177] The AR device (AR glasses) captures customer operation information. Specifically, it uses cameras and sensors to detect the customer's gaze and hand movements and collects that data. The input is the customer's gaze direction and hand gestures, and the output is data that compiles this operation information.

[0178] Step 2:

[0179] The AR device sends the captured operation information to the server. Specifically, the operation information is transferred to the server in real time via the network. The input is the operation information data obtained in step 1, and the output is the operation information data received by the server.

[0180] Step 3:

[0181] The server stores the received operation information in a data queue. Specifically, the server temporarily saves the operation information and prepares it for analysis. The input is the operation information data sent from the AR terminal, and the output is the operation information stored in the data queue.

[0182] Step 4:

[0183] The server's generative AI model analyzes the operation information. Specifically, the operation information is input into a generative AI model (e.g., GPT-4) to infer the customer's actions. In this process, hand movements and gaze patterns are recognized to infer specific actions (e.g., product selection). The input is the operation information stored in the data queue, and the output is the inferred action result.

[0184] Step 5:

[0185] The server generates feedback based on the inference results. Specifically, it generates feedback data (visual and tactile responses) according to the inferred actions and sends a notification to the remote store staff. The input is the inference results of the generative AI model, and the output is feedback data sent to the remote store staff and the VR gloves.

[0186] Step 6:

[0187] Remote store staff use VR gloves to virtually demonstrate to customers how to operate products. Specifically, they operate the products through the VR gloves and send the operation information to the server. The input is a feedback notification from the server, and the output is feedback data as a demonstration of the operation.

[0188] Step 7:

[0189] The server analyzes the operation information of the remote store staff and generates feedback. Specifically, it analyzes the operation information of the remote store staff and generates visual and tactile feedback to be displayed on the AR terminal (AR glasses). The input is the operation information from the remote store staff, and the output is feedback data sent to the AR glasses.

[0190] Step 8:

[0191] The AR device (AR glasses) provides the customer with the feedback received from the server. Specifically, it displays guides and arrows that visually show how to use the product, and provides tactile feedback as needed. The input is the feedback data from the server, and the output is the visual and tactile response that the customer actually experiences.

[0192] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0193] This invention provides a system for realizing interactions across different dimensions. This system consists of an AR terminal, a server, a VR terminal, and an emotion engine, and its main processing steps include capturing operation information, transmitting data, analyzing data, recognizing emotions, generating feedback, and applying feedback.

[0194] System configuration

[0195] AR device

[0196] AR devices are devices that use augmented reality technology to display digital information overlaid on the real world. These devices are equipped with cameras and sensors that capture the position and movement history of the AR user's hands and fingers in real time.

[0197] server

[0198] The server is a high-performance computer system that communicates with the AR and VR devices via a network. The server is equipped with a generative AI model and an emotion engine, and analyzes the operation information received from the AR device to estimate the user's actions and emotions.

[0199] VR device

[0200] A VR device is a device that uses virtual reality technology to provide an immersive virtual space. Its main components include a head-mounted display (HMD) and a vibration device. The VR device provides tactile or visual responses to the VR user based on feedback signals sent from the server.

[0201] Emotion Engine

[0202] The emotion engine is a software component that recognizes the emotions of AR users from their voices, facial expressions, gestures, etc. This allows it to generate more accurate feedback by combining it with operational information.

[0203] Program processing and examples

[0204] AR device operation capture

[0205] The AR device captures gestures made by the AR user toward the VR user's character. For example, if the AR user uses a tablet to make a gesture of patting the VR user's character's head through the screen, the sensor will capture the hand movement and position information.

[0206] Sending and Receiving Data

[0207] The AR device sends the captured operation information to the server, using a real-time communication protocol (e.g., WebSocket) to transmit data with low latency.

[0208] Data analysis and behavioral inference

[0209] The server uses the generated AI model to analyze the received operation information. Specifically, it infers the AR user's actions from hand movements and position information. In this process, it uses pre-trained gesture classes to classify actions such as "stroking" and "hitting."

[0210] emotion recognition

[0211] The server's emotion engine analyzes the AR user's voice, facial expressions, and gestures to recognize their emotions. For example, if the AR user is smiling and patting someone on the head, the emotion of "joy" is recognized.

[0212] Generate and send feedback

[0213] The server generates a feedback signal based on the analysis results and the recognized emotion. For example, if the action of "stroking" and the emotion of "joy" are inferred, the server generates an appropriate vibration pattern and visual feedback that conveys joy, and sends that signal to the VR device.

[0214] Applying Feedback

[0215] The VR device receives the feedback signal sent from the server. The received signal is used as an instruction to provide tactile and visual responses to the VR user. Specifically, the VR user's head-mounted device (HMD) generates vibrations, and animations of being petted or graphics conveying pleasure are displayed in the VR space.

[0216] Specific examples

[0217] For example, suppose an AR user uses a tablet to make a gesture of patting a VR user's head while smiling. The AR device captures this gesture and facial expression and sends it to the server. The server's generative AI model analyzes this data to infer the action of "patting," and the emotion engine recognizes the emotion of "joy." The server then generates vibration and visual feedback to convey joy and sends it to the VR device. The VR device receives this feedback and generates a slight vibration in the VR user's HMD, while simultaneously displaying a graphic that expresses joy along with an animation of the pat. This allows the VR user to experience the sensation of actually being petted while also sensing the AR user's emotions.

[0218] This system allows AR and VR users to interact intuitively and emotionally, even though they are in different dimensions.

[0219] The processing flow will be explained below.

[0220] Step 1:

[0221] The AR device captures the gestures made by the AR user. Specifically, the tablet's camera and sensors capture the AR user's hand position (x, y, z coordinates) and movement (velocity, acceleration) in real time. This information is captured frame by frame.

[0222] Step 2:

[0223] The AR device sends the acquired gesture data to the server using a real-time communication protocol (e.g., WebSocket) to transmit the data with low latency.

[0224] Step 3:

[0225] The server receives the gesture data sent from the AR device and temporarily stores the received data in a queue.

[0226] Step 4:

[0227] The server uses an emotion engine to recognize the emotions of the AR user. For example, the server analyzes the voice and facial expression data received from the AR device to identify emotions such as "joy" or "sadness."

[0228] Step 5:

[0229] The server analyzes the received gesture data using a generative AI model to infer the AR user's actions. For example, it can infer the action of "stroking" from hand movements and position information.

[0230] Step 6:

[0231] The server generates a feedback signal based on the analysis results and the recognized emotion. For example, if the action of "stroking" and the emotion of "joy" are inferred, the server generates a specific vibration pattern and visual feedback that conveys joy.

[0232] Step 7:

[0233] The server transmits the generated feedback signal to the VR device using a real-time communication protocol with low latency.

[0234] Step 8:

[0235] The VR device receives the feedback signals sent from the server, which are used to provide haptic and visual responses.

[0236] Step 9:

[0237] The VR device applies a reaction to the VR user based on the received feedback signal. Specifically, it generates vibrations in the VR user's HMD and displays animations of being petted and graphics that convey pleasure in the VR space.

[0238] Step 10:

[0239] The VR user feels the AR user stroking them through vibrations in the head and visual feedback, and can also sense the AR user's emotions based on the feedback.

[0240] As a specific example, consider a scene in which an AR user uses a tablet to make a gesture of patting a VR user's head, smiling while doing so. The AR device captures the gesture and facial expression and sends the data to the server. The server's emotion engine recognizes the smile as the emotion of "happiness," and the generative AI model infers the action of "patting." The server then generates a feedback signal based on the emotion and action and sends it to the VR device. The VR device receives this feedback, generates a slight vibration in the VR user's HMD, and displays an animation expressing happiness. This gives the VR user the feeling of actually being petted, and also allows them to sense the AR user's emotion of happiness.

[0241] Through these steps, the present invention realizes intuitive and emotionally rich interaction that transcends the dimensions of AR and VR.

[0242] Example 2

[0243] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0244] In conventional AR and VR systems, it has been difficult to achieve intuitive and emotion-based interaction across different dimensions. In particular, there has been a lack of systems that can analyze users' gestures and actions in real time and provide appropriate feedback based on that analysis. In addition, there are limited means to accurately recognize users' emotions and generate feedback based on those emotions.

[0245] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0246] In this invention, the server includes a means for transmitting operation information acquired from the AR terminal to the server, a means including a generative AI model for analyzing the operation information and inferring an action, an emotion recognition means for analyzing the operation information and the user's facial expressions and voice and recognizing emotions, a means for generating feedback based on the inferred action and the recognized emotion and transmitting it to the VR terminal, and a means for receiving the feedback and providing a tactile or visual response to the VR user, thereby enabling intuitive and emotional interactions across different dimensions.

[0247] An "AR terminal" is a device that uses augmented reality technology to overlay digital information on the real world.

[0248] "Operation information" is data on the movements and positions of the user's hands and fingers acquired from the AR terminal.

[0249] A "server" is a high-performance computer system that communicates with AR and VR terminals to analyze and process data.

[0250] A "generative AI model" is a model that uses machine learning algorithms to analyze user gesture data and classify and infer specific actions.

[0251] An "emotion recognition means" is a system or algorithm that analyzes the user's voice, facial expressions, gestures, etc. to recognize their emotions.

[0252] "Feedback" refers to the tactile or visual response generated based on actions and perceived emotions.

[0253] A "VR terminal" is a device that uses virtual reality technology to provide an immersive virtual space, and includes a head-mounted display (HMD) and a vibration device.

[0254] "Haptic feedback" refers to a physical stimulus (e.g., vibration) that provides a tactile sensation to the user.

[0255] "Visual response" refers to providing feedback to the user through visual information, including animations and graphics.

[0256] "Interaction across different dimensions" refers to users in the real world and the virtual world sharing and interacting with each other's actions and emotions.

[0257] The system of this invention is composed of an AR terminal, a server, a VR terminal, and emotion recognition means. We will explain a specific embodiment that realizes intuitive and emotional interaction between AR users and VR users across different dimensions through the operation of the entire system.

[0258] Hardware and software used

[0259] AR terminal: An augmented reality device such as Microsoft (registered trademark) HoloLens (registered trademark). Equipped with a camera and sensors, it captures the user's movements in real time.

[0260] Server: A high-performance computer system (e.g., AWS EC2 instance) that runs the generative AI model and emotion recognition method.

[0261] VR device: A virtual reality device such as the Oculus Rift, including a head-mounted display (HMD) and a vibration device.

[0262] Emotion recognition means: Uses Microsoft Azure (registered trademark) emotion API, etc.

[0263] Data processing and calculation

[0264] When a user uses an AR device to make a gesture (e.g., stroking the head) to the VR user's character, the camera and sensors of the AR device capture this gesture and obtain the position information and movement history of the hand and fingers in real time.

[0265] The operation information acquired by the AR device is sent to the server using a real-time communication protocol (e.g., WebSocket). The server receives this data and begins analyzing it using the generative AI model.

[0266] The server uses a generated AI model (for example, a custom model based on TENSORFLOW (registered trademark)) to analyze hand movements and position information and infer actions such as "stroking." Along with the analyzed action data, the server captures the user's facial expressions and voice, and recognizes their emotions using emotion recognition means.

[0267] For example, if the user is smiling and stroking the user's head, the emotion recognition means recognizes the emotion as "joy." The server generates a feedback signal based on the analysis of the user's behavior and emotion.

[0268] The feedback signals include haptic feedback (e.g., vibration of the HMD) and visual feedback (e.g., animation of being petted and graphics expressing pleasure). The server generates the feedback signals and sends them to the VR device.

[0269] The VR device receives the feedback signal sent from the server and provides specific tactile and visual responses to the VR user, such as generating a slight vibration in the HMD and displaying a graphic indicating pleasure along with an animation of the user being stroked.

[0270] Specific examples

[0271] For example, suppose a user uses a tablet to make a gesture of patting a VR user's head while smiling. The AR device captures this gesture and facial expression and sends it to the server. The server's generative AI model analyzes this data to infer the action of "patting," and the emotion recognition means recognizes the emotion of "joy." The server then generates vibration and visual feedback to convey joy and sends it to the VR device. The VR device receives this feedback and generates a slight vibration in the HMD, while simultaneously displaying a graphic that expresses joy along with an animation of the user being stroked. This allows the VR user to experience the sensation of actually being stroked while also sensing the user's emotions.

[0272] Prompt Sentence Examples

[0273] "Imagine an AR user using a tablet to smile and stroke the head of the VR user character through the screen. The AR device captures this gesture and facial expression and sends it to the server in real time. The server's AI model analyzes it and detects the 'stroking' gesture, and the emotion recognition means recognizes the emotion 'happiness'. As a result, the server generates appropriate haptic and visual feedback and sends it to the VR device. The VR device receives this feedback and vibrates the HMD, displaying an animation of the user being stroked and a graphic of happiness."

[0274] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0275] Step 1:

[0276] The user makes a gesture using the AR device. Specifically, the user uses a tablet to pat the head of the VR user's character. At this time, the camera and sensors of the AR device capture the position information and movement history of the hand and fingers in real time. The input is the user's gesture, and the output is real-time position information and movement history.

[0277] Step 2:

[0278] The operation information captured by the device (AR device) is sent to the server via a real-time communication protocol (e.g., WebSocket). Specifically, the AR device forms the position information and movement history of the hands and fingers it has acquired into a data packet and sends it to the server. The input is the real-time position information and movement history, and the output is the sent data packet.

[0279] Step 3:

[0280] The server receives data packets sent from the AR device. Specifically, the server receives the data packets via WebSocket, decodes the packets, and extracts the operation information. The input is the sent data packet, and the output is the operation information.

[0281] Step 4:

[0282] The operation information received by the server is analyzed using a generative AI model. Specifically, the generative AI model (e.g., a custom model based on TensorFlow) analyzes the position information and movement history of the hand and fingers, and infers that the user's gesture is a specific action, such as "stroking." This analysis is performed based on pre-trained gesture classes. The input is operation information, and the output is data on the inferred action.

[0283] Step 5:

[0284] The server uses an emotion recognition means to recognize the user's emotions. Specifically, it analyzes the user's voice, facial expression, and gesture information, and the emotion recognition means (e.g., Microsoft Azure's Emotion API) detects emotions such as "joy." The input is voice, facial expression, and gesture information, and the output is recognized emotion data.

[0285] Step 6:

[0286] The server generates a feedback signal based on the estimated action and the recognized emotion. Specifically, the server combines the action of "stroking" with the emotion of "joy" to generate a signal including haptic feedback (e.g., HMD vibration) and visual feedback (e.g., an animation of being stroked and a graphic expressing joy). The input is action data and emotion data, and the output is a feedback signal.

[0287] Step 7:

[0288] The server generates feedback signals and sends them to the VR device. Specifically, the feedback signals are formed as data packets and sent to the VR device via WebSocket. This provides sensory information to the VR user. The input is the generated feedback signal, and the output is the transmitted data packet.

[0289] Step 8:

[0290] The terminal (VR terminal) receives the feedback signal sent from the server and provides specific tactile and visual responses. Specifically, it generates a slight vibration in the HMD and displays an animation of being stroked and a graphic that expresses pleasure in the VR space. The received feedback signal is the input, and the tactile and visual feedback is provided as the output.

[0291] (Application example 2)

[0292] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0293] It solves the challenge of realizing real-time, emotional interaction between different dimensions, namely augmented reality (AR) and virtual reality (VR), and also improves the customer experience in virtual stores, enabling personalized shopping experiences.

[0294] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means including a generative artificial intelligence model that analyzes operation information acquired from the AR terminal and infers an action, means for generating feedback based on the inferred action and the recognized emotion and transmitting the feedback to the VR terminal, and means for the feedback to include visual and tactile feedback that personalizes the shopping experience in the virtual space. This enables intuitive and emotionally rich interactions between different dimensions and makes it possible to provide a personalized customer experience in the virtual store.

[0295] "Different dimensions" refers to the different realms of existence of augmented reality (AR), which overlays digital information on the real world, and virtual reality (VR), which provides a virtual space.

[0296] A "generative artificial intelligence model" is a modeling method based on machine learning and deep learning that is used to analyze received operational information and infer actions and emotions.

[0297] "Operation information" refers to data such as hand and finger position information and movement history obtained from the AR device.

[0298] "Feedback" refers to a reaction signal generated based on the inferred action and the recognized emotion, to provide the user with a tactile or visual response.

[0299] "Virtual space" refers to a virtual environment or world that can be experienced through a VR device.

[0300] "Gesture data" refers to data related to hand movements and gestures made by a user.

[0301] "Facial expression data" refers to data relating to the user's facial expressions.

[0302] "Promotion information" is information related to sales and special campaigns for specific products.

[0303] "Visual feedback" refers to information provided to a user through their sense of sight, including graphics and animations displayed on a display.

[0304] An "AR terminal" refers to a device that uses augmented reality technology to overlay digital information on the real world.

[0305] A "VR terminal" refers to a device that uses virtual reality technology to provide an immersive virtual space.

[0306] System configuration

[0307] A system for realizing this application example uses the following hardware and software.

[0308] Hardware

[0309] 1. AR device: A device (e.g., smart glasses) that uses augmented reality technology to overlay digital information on the real world.

[0310] 2. VR terminal: A device that uses virtual reality technology to provide an immersive virtual space (e.g., a head-mounted display).

[0311] 3. Server: A high performance computer server.

[0312] software

[0313] 1. Generative AI model: An artificial intelligence model (e.g., TensorFlow or PyTorch) for analyzing hand movements and facial expressions.

[0314] 2. Emotion engine: Software that recognizes emotions by analyzing voice and facial expression data (e.g., Google® Cloud Natural Language API, Amazon Comprehend).

[0315] 3. Communication protocol: A protocol for communicating data in real time (e.g., WebSocket).

[0316] What the program does

[0317] This system processes data and realizes interactions in the following steps:

[0318] AR device processing

[0319] AR devices are equipped with sensors to capture the user's hand and finger movements and facial expressions. In particular, the AR device acquires the captured gesture and facial expression data in real time and transmits it to a server. For example, if a user points at a product using smart glasses and their facial expression shows delight, that information will be captured.

[0320] Server Processing

[0321] The server is equipped with a generative AI model and emotion engine. The server analyzes the operation information sent from the AR device and estimates the action and emotion. Specifically, it analyzes hand movements and position information to estimate the "pointing" action and further recognizes the emotion of "happiness" from the user's facial expression.

[0322] Generate and send feedback

[0323] Based on the analysis results, the server generates appropriate feedback, including visual and tactile responses. For example, if the estimated action is "pointing" and the emotion is "joy," the server generates visual feedback containing detailed product information and promotional information and sends it to the VR device.

[0324] Applying Feedback

[0325] The VR device responds appropriately in the virtual space based on the received feedback signal, providing visual feedback to the user by displaying product guides or spontaneously displaying promotional information about specific products.

[0326] Examples and prompts

[0327] For example, if a customer uses smart glasses to look at a product in a virtual store, makes a gesture of touching it with their hands, and shows an excited expression, this information is captured and sent to the server. The server analyzes this data and recognizes the behavior of "showing interest in a particular product" and the emotion of "excitement." It then generates feedback on the VR device that displays detailed product information and special discount promotions. The example prompt sentences used by the generative AI model are as follows:

[0328] User is viewing and interacting with a product. Capture the movements and facial expressions, and provide appropriate promotional feedback.

[0329] This makes the user's shopping experience in a virtual store more intuitive and emotionally satisfying.

[0330] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0331] Step 1:

[0332] The AR device captures the user's hand and finger movements and facial expressions in real time. Gesture and facial expression data obtained from sensors is received as input and sent to a server. Specifically, the smart glasses' cameras and sensors capture and measure the user's movements, and the data is processed and compiled as operation information. Operation information includes the position and movement of the hands, facial expressions, etc. This information is sent to the server in real time via WebSocket.

[0333] Step 2:

[0334] The server receives operation information sent from the AR device. The received data includes the user's gestures and facial expression data. The server analyzes this data using a generative AI model and an emotion engine. Specifically, the generative AI model analyzes hand movements and position information to infer actions, and the emotion engine analyzes facial expression data and voice to recognize emotions. As a result of this analysis, the server outputs an action estimation result (e.g., "pointing") and an emotion recognition result (e.g., "excitement").

[0335] Step 3:

[0336] The server generates appropriate feedback based on the estimated action and the recognized emotion. It receives the action estimation results and emotion recognition results as input and generates feedback data accordingly. For example, if the action of "pointing" and the emotion of "excitement" are recognized, the server visually displays detailed information about a product and creates feedback including promotional information related to that product. The generated feedback data is sent to the VR device.

[0337] Step 4:

[0338] The VR device receives the feedback data sent from the server. The received data includes visual feedback and haptic feedback. Specifically, the VR device displays detailed product information and promotional information in the virtual space as visual feedback, and generates device vibrations and other haptic feedback. This process allows the user to experience real-time feedback in the virtual space.

[0339] Step 5:

[0340] Users experience feedback provided through the VR device. Specifically, when a user points at a product in the virtual space, detailed information about that product is displayed. Also, if a user shows interest in a particular product, related promotional information is displayed. This makes the shopping experience in the virtual store more intuitive and emotionally satisfying.

[0341] The above steps enable interaction across different dimensions and improve customer experience in virtual stores.

[0342] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0343] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0344] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0345] [Second embodiment]

[0346] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0347] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0348] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0349] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0350] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0351] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0352] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0353] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0354] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0355] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0356] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0357] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0358] This invention provides a system for realizing interactions between different dimensions. This system consists of an AR terminal, a server, and a VR terminal, and its main processing steps include capturing operation information, transmitting data, analyzing data, generating feedback, and applying feedback.

[0359] System configuration

[0360] AR device

[0361] AR devices are devices that use augmented reality technology to display digital information overlaid on the real world. These devices are equipped with cameras and sensors that capture the position and movement history of the AR user's hands and fingers in real time.

[0362] server

[0363] The server is a high-performance computer system that communicates with the AR and VR devices via a network. The server is equipped with a generative AI model that analyzes the operation information received from the AR device and infers the user's actions.

[0364] VR device

[0365] A VR device is a device that uses virtual reality technology to provide an immersive virtual space. Its main components include a head-mounted display (HMD) and a vibration device. The VR device provides tactile or visual responses to the VR user based on feedback signals sent from the server.

[0366] Program processing and examples

[0367] AR device operation capture

[0368] The AR device captures gestures made by the AR user toward the VR user's character. For example, if the AR user uses a tablet to make a gesture of patting the VR user's character's head through the screen, the sensor will capture the hand movement and position information.

[0369] Sending and Receiving Data

[0370] The AR device transmits the captured operation information to the server, which receives the operation information and stores it in a data queue.

[0371] Data analysis and behavioral inference

[0372] The server uses the generative AI model to analyze the received operation information. For example, it can infer the action of "stroking" based on the hand's trajectory, speed, and movement pattern. The generative AI model then classifies the action using gesture classes it has learned in advance.

[0373] Generate and send feedback

[0374] The server generates feedback based on the analysis results. For example, if it detects a "stroking" action, it generates an appropriate vibration pattern or haptic feedback and sends that signal to the VR device.

[0375] Applying Feedback

[0376] The VR device provides a response to the VR user based on the feedback signal received from the server. Specifically, the VR user's head-mounted device (HMD) generates a slight vibration to recreate the feeling of being stroked. Additionally, an animation of the user being stroked is displayed in the VR space as visual feedback.

[0377] Specific examples

[0378] For example, suppose an AR user uses a tablet to make a gesture of stroking the VR user's head. The AR device captures this gesture and sends it to the server. The server's generative AI model analyzes this data and infers the action of "stroking." The server then generates an appropriate vibration pattern and sends it to the VR device. The VR device receives this feedback and generates a slight vibration in the VR user's HMD while simultaneously displaying an animation of the stroking. This allows the VR user to experience the sensation of actually being stroked.

[0379] This system allows AR and VR users to communicate intuitively and effectively, even though they are in different dimensions.

[0380] The processing flow will be explained below.

[0381] Step 1:

[0382] The AR device captures the gestures made by the AR user. Specifically, the tablet's camera and sensors capture the AR user's hand position (x, y, z coordinates) and movement (velocity, acceleration) in real time. This information is captured frame by frame.

[0383] Step 2:

[0384] The AR device sends the captured operation information to the server, which uses a real-time communication protocol (e.g., WebSocket) to transmit data with low latency.

[0385] Step 3:

[0386] The server receives the operation information sent from the AR device. The received data is temporarily stored in a queue and prepared for analysis by the generative AI model.

[0387] Step 4:

[0388] The server uses the generative AI model to analyze the received operation information. Specifically, it infers the AR user's actions from hand movements and position information. In this process, it uses pre-trained gesture classes to classify actions such as "stroking" and "hitting."

[0389] Step 5:

[0390] The server generates feedback signals based on the analysis results. For example, if the action of "stroking" is detected, it generates appropriate vibration patterns or visual feedback to reproduce the sensation of stroking.

[0391] Step 6:

[0392] The server transmits the generated feedback signals to the VR device, again using a real-time communication protocol to transmit the signals with low latency.

[0393] Step 7:

[0394] The VR device receives the feedback signal sent from the server, which is used as an instruction to provide tactile and visual responses to the VR user.

[0395] Step 8:

[0396] The VR device applies feedback by vibrating the VR user's head-mounted device (HMD) and also provides visual feedback by displaying an animation of the object being stroked in the VR space.

[0397] Step 9:

[0398] The VR user feels the vibrations in their head and visual feedback, allowing them to feel as if the AR user is stroking them, creating a mutual interaction experience that transcends the dimensions of AR and VR.

[0399] Example 1

[0400] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0401] Conventional virtual reality (VR) and augmented reality (AR) systems have the problem of not intuitively reflecting interactions between users in real time. In particular, it is difficult to provide appropriate tactile and visual feedback when multiple users in different dimensions participate in simultaneous communication. Furthermore, accurate recognition of gestures and the quality of feedback can be reduced, resulting in a poor user experience.

[0402] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0403] In this invention, the server includes a means for transmitting operation information acquired from the AR terminal to the server, a means including a generative AI model for analyzing the operation information and inferring an action, and a means for generating feedback based on the inferred action and transmitting the feedback to the VR terminal, thereby enabling intuitive and effective interaction even between users in different dimensions.

[0404] An "AR terminal" is a device that uses augmented reality technology to overlay digital information onto the real world.

[0405] A "server" is a high-performance computer system that communicates with AR and VR terminals via a network.

[0406] "Operation information" is data related to the user's movements and gestures, and is information captured in real time by cameras and sensors.

[0407] A "generative AI model" is a machine learning model that analyzes received operation information and infers user actions, and uses pre-trained gesture classes.

[0408] "Feedback" refers to reaction data generated based on the analysis results, and includes tactile and visual reactions.

[0409] A "VR terminal" is a device that uses virtual reality technology to provide an immersive virtual space, and includes a head-mounted display and a vibration device.

[0410] "Tactile feedback" refers to feedback that provides a direct sensation to the user's body, and includes physical stimuli such as vibrations.

[0411] "Visual response" refers to providing direct feedback to the user's vision, such as on-screen animation.

[0412] A "gesture" is an operation based on the movement of a user's hands or fingers, and is a motion pattern that indicates a specific action.

[0413] A "network" is an infrastructure for realizing communication between a server and a terminal, and includes the Internet and a local area network (LAN).

[0414] A "head-mounted display (HMD)" is a display device worn by the user on the head, providing an immersive VR experience.

[0415] This invention is a system for realizing interactions between different dimensions, and is mainly composed of an AR terminal, a server, and a VR terminal. Each component and its function will be explained in detail below.

[0416] AR device

[0417] AR devices are devices that use augmented reality technology to overlay digital information onto the real world. Specifically, they are equipped with cameras and sensors and have the ability to capture the positional information and movement history of the user's hands and fingers in real time. For example, an AR-enabled tablet can be used, and it can accurately capture the gestures the user makes with a virtual character.

[0418] server

[0419] The server is a high-performance computer system that communicates with the AR and VR devices via a network. It contains a generative AI model that analyzes the operation information sent from the AR device and infers the user's actions. The generative AI model classifies the actions using pre-trained gesture classes and sends the results to the feedback generation module.

[0420] VR device

[0421] A VR device is a device that uses virtual reality technology to provide an immersive virtual space. Its main components are a head-mounted display (HMD) and a vibration device, and it has the function of providing tactile and visual responses to the VR user based on feedback signals sent from the server. Specifically, the VR user's HMD vibrates to reproduce the feeling of being stroked, and displays a stroke animation as visual feedback.

[0422] Specific examples

[0423] For example, if an AR user uses a tablet to make a gesture of stroking the head of a VR user through the screen, the process proceeds as follows: The user operates the AR device, and their hand movements are captured by the camera and sensors. The captured operation information is sent to the server via the network, and the server's generative AI model analyzes the data and infers the action of "stroking." The server then generates an appropriate vibration pattern and visual feedback, which it sends to the VR device. The VR device receives the feedback signal, generates a slight vibration in the HMD, and simultaneously displays an animation of the user being stroked.

[0424] Prompt Sentence Examples

[0425] Specifically, the following prompts are used:

[0426] "Analyze the following operation information to generate feedback as if the VR user's character is being stroked: hand movement trajectory, speed, and movement pattern: {hand movement data}"

[0427] This system enables AR and VR users to communicate intuitively and effectively in real time, even though they are in different dimensions.

[0428] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0429] Step 1:

[0430] Operation capture using an AR device

[0431] Input: User gestures

[0432] Output: Hand position information, movement history

[0433] Specific behavior:

[0434] The user operates the AR device and touches a virtual object (e.g., the head of a VR character) through the screen. The camera and sensors on the AR device capture this action in real time. Specifically, the position information and movement history of the hands and fingers are acquired and converted into data.

[0435] Step 2:

[0436] Sending data and receiving it on the server

[0437] Input: Hand position information, movement history

[0438] Output: Sending operation information data, storing received data

[0439] Specific behavior:

[0440] The AR device sends the hand position information and movement history to the server. The data is transferred via the network, and the server receives the data and temporarily stores it in a data queue.

[0441] Step 3:

[0442] Data analysis and behavioral inference

[0443] Input: Operation information data

[0444] Output: Estimated result of the action

[0445] Specific behavior:

[0446] The server uses the generated AI model to analyze the received operation information data. Specifically, it analyzes the trajectory, speed, and movement patterns of the hand movement to infer actions such as "stroking." For example, from an analysis of the smooth movement of the hand and its speed, it can infer with a high degree of probability that the action is "stroking."

[0447] Step 4:

[0448] Generate and send feedback

[0449] Input: Estimated result of the action

[0450] Output: Feedback data

[0451] Specific behavior:

[0452] The server generates appropriate feedback based on the action estimation. For example, a vibration pattern or visual feedback (animation) corresponding to the action of "stroking" is generated. The generated feedback data is sent to the VR device.

[0453] Step 5:

[0454] Applying Feedback

[0455] Input: Feedback data

[0456] Output: Tactile and visual feedback to the user

[0457] Specific behavior:

[0458] The VR device responds to the user based on the feedback data received from the server. The HMD generates appropriate vibrations and displays animations of the character being stroked in the VR space. For example, the visual effect of a character's head being stroked along with a slight vibration gives the user the sensation of actually stroking the character.

[0459] (Application example 1)

[0460] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0461] While the direct interaction between customers and store staff forms part of the value of the shopping experience in a physical store, there are limitations in that not all customers can receive support at the same time. Another issue is that when store staff provide support remotely, the information and experience customers receive is limited. This can lead to customers not being able to fully understand how to use a product or its features.

[0462] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0463] In this invention, the server includes: means for transmitting operation information acquired from the AR device to the server; means including a generative AI model that analyzes the operation information and infers actions; means for generating feedback based on the inferred actions and transmitting it to the VR device; means for receiving the feedback and providing tactile or visual feedback to the VR user; means for a customer to wear AR glasses in a physical store and for a remote store staff member to provide product operation instructions using VR gloves; and means for the server to analyze the customer's behavior, generate appropriate feedback, and display it on the AR glasses. This allows customers to intuitively and effectively understand how to use and use products while providing valuable remote support.

[0464] "Interaction between different dimensions" refers to the interaction and information exchange between users that takes place between a virtual space, which is different from the physical space, and a real space using virtual reality or augmented reality.

[0465] An "AR terminal" is a device that uses augmented reality technology to overlay digital information on the real world, and is equipped with a camera and sensors.

[0466] "Operation information" refers to data acquired by the AR device when it recognizes operations such as user movements and gestures.

[0467] A "server" is a high-performance computer system that communicates with various terminals via a network and analyzes data.

[0468] A "generative AI model" is an algorithm or model that uses artificial intelligence to analyze data and infer user operations and behavior.

[0469] A "VR terminal" is a device that uses virtual reality technology to provide users with an immersive virtual space, and includes head-mounted displays and haptic devices.

[0470] "Feedback" refers to the tactile or visual response that is generated in response to a user's operation or action.

[0471] A "brick and mortar store" is a commercial establishment that exists in a physical location and can be visited in person by customers.

[0472] "AR glasses" are glasses-type devices that use augmented reality technology, allowing users to visually view digital information overlaid on the real world.

[0473] "Remote store staff" refers to store staff who are physically located in a different location from the physical store but who support customers virtually.

[0474] "VR gloves" are glove-shaped devices that allow users to operate the device with their hands in a virtual reality world and provide haptic feedback.

[0475] "Analyzing customer behavior" means collecting data on the operations and actions performed by customers, and analyzing that data to infer specific actions.

[0476] This invention provides a system that uses an AR terminal, a server, and a VR terminal to improve the shopping experience in a physical store. This system allows customers to wear AR glasses in the physical store, and remote store staff can use VR gloves to provide instructions on how to operate products.

[0477] First, a customer wears AR glasses and walks around a physical store. The AR device (e.g., a glasses-type device that uses augmented reality technology) captures customer operation information through cameras and sensors. For example, when a customer looks at a product or makes a gesture of reaching out, that operation information is acquired in real time.

[0478] This operation information is then sent to a server and stored in a data queue. The server is a high-performance computer system (e.g., an AWS EC2 instance) that analyzes the received operation information using a generative AI model (e.g., OpenAI's GPT-4). Specifically, the server analyzes hand movements, gaze direction, and movement patterns to infer the action of "selecting a product."

[0479] Based on the analyzed actions, the server generates feedback. A remote store staff member wearing VR gloves (e.g., a glove-type device for operating with hands in a virtual reality world) can use the feedback to demonstrate to the customer how to operate the product and its characteristics. The server generates appropriate feedback (visual and tactile responses) and displays it on the AR glasses, allowing the customer to experience the store staff as if they were right in front of them.

[0480] Here is one concrete example:

[0481] 1. A customer puts on AR glasses in a physical store and looks at a specific product.

[0482] 2. The AR device captures the eye movements and hand gestures and sends the operation information to the server.

[0483] 3. The server's generated AI model analyzes the operation information and infers the action of "selecting a product."

[0484] 4. The server sends notifications to remote store staff based on the analysis results.

[0485] 5. Remote store staff will use VR gloves to virtually demonstrate how to operate the product.

[0486] 6. Feedback is displayed on the customer's AR glasses, allowing them to visually confirm how to use the product and its features.

[0487] Example prompts for generative AI models:

[0488] Analyze the following input data and infer the actions taken by the customer (product selection, inquiry, etc.).

[0489] Input data: { "Action": "Reach out", "Target": "Product code 12345", "Timestamp": "2023-10-01T08:00:00Z"}

[0490] Please output the analysis results in the following format:

[0491] {

[0492] "Action": "Product Selection",

[0493] "Product Code": "12345",

[0494] "Recommended feedback": "View a demonstration of how to use the product"

[0495] }

[0496] The invention makes the in-store shopping experience more intuitive and effective, allowing customers to understand the usage and characteristics of products with real-time support from remote store staff.

[0497] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0498] Step 1:

[0499] The AR device (AR glasses) captures customer operation information. Specifically, it uses cameras and sensors to detect the customer's gaze and hand movements and collects that data. The input is the customer's gaze direction and hand gestures, and the output is data that compiles this operation information.

[0500] Step 2:

[0501] The AR device sends the captured operation information to the server. Specifically, the operation information is transferred to the server in real time via the network. The input is the operation information data obtained in step 1, and the output is the operation information data received by the server.

[0502] Step 3:

[0503] The server stores the received operation information in a data queue. Specifically, the server temporarily saves the operation information and prepares it for analysis. The input is the operation information data sent from the AR terminal, and the output is the operation information stored in the data queue.

[0504] Step 4:

[0505] The server's generative AI model analyzes the operation information. Specifically, the operation information is input into a generative AI model (e.g., GPT-4) to infer the customer's actions. In this process, hand movements and gaze patterns are recognized to infer specific actions (e.g., product selection). The input is the operation information stored in the data queue, and the output is the inferred action result.

[0506] Step 5:

[0507] The server generates feedback based on the inference results. Specifically, it generates feedback data (visual and tactile responses) according to the inferred actions and sends a notification to the remote store staff. The input is the inference results of the generative AI model, and the output is feedback data sent to the remote store staff and the VR gloves.

[0508] Step 6:

[0509] Remote store staff use VR gloves to virtually demonstrate to customers how to operate products. Specifically, they operate the products through the VR gloves and send the operation information to the server. The input is a feedback notification from the server, and the output is feedback data as a demonstration of the operation.

[0510] Step 7:

[0511] The server analyzes the operation information of the remote store staff and generates feedback. Specifically, it analyzes the operation information of the remote store staff and generates visual and tactile feedback to be displayed on the AR terminal (AR glasses). The input is the operation information from the remote store staff, and the output is feedback data sent to the AR glasses.

[0512] Step 8:

[0513] The AR device (AR glasses) provides the customer with the feedback received from the server. Specifically, it displays guides and arrows that visually show how to use the product, and provides tactile feedback as needed. The input is the feedback data from the server, and the output is the visual and tactile response that the customer actually experiences.

[0514] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0515] This invention provides a system for realizing interactions across different dimensions. This system consists of an AR terminal, a server, a VR terminal, and an emotion engine, and its main processing steps include capturing operation information, transmitting data, analyzing data, recognizing emotions, generating feedback, and applying feedback.

[0516] System configuration

[0517] AR device

[0518] AR devices are devices that use augmented reality technology to display digital information overlaid on the real world. These devices are equipped with cameras and sensors that capture the position and movement history of the AR user's hands and fingers in real time.

[0519] server

[0520] The server is a high-performance computer system that communicates with the AR and VR devices via a network. The server is equipped with a generative AI model and an emotion engine, and analyzes the operation information received from the AR device to estimate the user's actions and emotions.

[0521] VR device

[0522] A VR device is a device that uses virtual reality technology to provide an immersive virtual space. Its main components include a head-mounted display (HMD) and a vibration device. The VR device provides tactile or visual responses to the VR user based on feedback signals sent from the server.

[0523] Emotion Engine

[0524] The emotion engine is a software component that recognizes the emotions of AR users from their voices, facial expressions, gestures, etc. This allows it to generate more accurate feedback by combining it with operational information.

[0525] Program processing and examples

[0526] AR device operation capture

[0527] The AR device captures gestures made by the AR user toward the VR user's character. For example, if the AR user uses a tablet to make a gesture of patting the VR user's character's head through the screen, the sensor will capture the hand movement and position information.

[0528] Sending and Receiving Data

[0529] The AR device sends the captured operation information to the server, using a real-time communication protocol (e.g., WebSocket) to transmit data with low latency.

[0530] Data analysis and behavioral inference

[0531] The server uses the generated AI model to analyze the received operation information. Specifically, it infers the AR user's actions from hand movements and position information. In this process, it uses pre-trained gesture classes to classify actions such as "stroking" and "hitting."

[0532] emotion recognition

[0533] The server's emotion engine analyzes the AR user's voice, facial expressions, and gestures to recognize their emotions. For example, if the AR user is smiling and patting someone on the head, the emotion of "joy" is recognized.

[0534] Generate and send feedback

[0535] The server generates a feedback signal based on the analysis results and the recognized emotion. For example, if the action of "stroking" and the emotion of "joy" are inferred, the server generates an appropriate vibration pattern and visual feedback that conveys joy, and sends that signal to the VR device.

[0536] Applying Feedback

[0537] The VR device receives the feedback signal sent from the server. The received signal is used as an instruction to provide tactile and visual responses to the VR user. Specifically, the VR user's head-mounted device (HMD) generates vibrations, and animations of being petted or graphics conveying pleasure are displayed in the VR space.

[0538] Specific examples

[0539] For example, suppose an AR user uses a tablet to make a gesture of patting a VR user's head while smiling. The AR device captures this gesture and facial expression and sends it to the server. The server's generative AI model analyzes this data to infer the action of "patting," and the emotion engine recognizes the emotion of "joy." The server then generates vibration and visual feedback to convey joy and sends it to the VR device. The VR device receives this feedback and generates a slight vibration in the VR user's HMD, while simultaneously displaying a graphic that expresses joy along with an animation of the pat. This allows the VR user to experience the sensation of actually being petted while also sensing the AR user's emotions.

[0540] This system allows AR and VR users to interact intuitively and emotionally, even though they are in different dimensions.

[0541] The processing flow will be explained below.

[0542] Step 1:

[0543] The AR device captures the gestures made by the AR user. Specifically, the tablet's camera and sensors capture the AR user's hand position (x, y, z coordinates) and movement (velocity, acceleration) in real time. This information is captured frame by frame.

[0544] Step 2:

[0545] The AR device sends the acquired gesture data to the server using a real-time communication protocol (e.g., WebSocket) to transmit the data with low latency.

[0546] Step 3:

[0547] The server receives the gesture data sent from the AR device and temporarily stores the received data in a queue.

[0548] Step 4:

[0549] The server uses an emotion engine to recognize the emotions of the AR user. For example, the server analyzes the voice and facial expression data received from the AR device to identify emotions such as "joy" or "sadness."

[0550] Step 5:

[0551] The server analyzes the received gesture data using a generative AI model to infer the AR user's actions. For example, it can infer the action of "stroking" from hand movements and position information.

[0552] Step 6:

[0553] The server generates a feedback signal based on the analysis results and the recognized emotion. For example, if the action of "stroking" and the emotion of "joy" are inferred, the server generates a specific vibration pattern and visual feedback that conveys joy.

[0554] Step 7:

[0555] The server transmits the generated feedback signal to the VR device using a real-time communication protocol with low latency.

[0556] Step 8:

[0557] The VR device receives the feedback signals sent from the server, which are used to provide haptic and visual responses.

[0558] Step 9:

[0559] The VR device applies a reaction to the VR user based on the received feedback signal. Specifically, it generates vibrations in the VR user's HMD and displays animations of being petted and graphics that convey pleasure in the VR space.

[0560] Step 10:

[0561] The VR user feels the AR user stroking them through vibrations in the head and visual feedback, and can also sense the AR user's emotions based on the feedback.

[0562] As a specific example, consider a scene in which an AR user uses a tablet to make a gesture of patting a VR user's head, smiling while doing so. The AR device captures the gesture and facial expression and sends the data to the server. The server's emotion engine recognizes the smile as the emotion of "happiness," and the generative AI model infers the action of "patting." The server then generates a feedback signal based on the emotion and action and sends it to the VR device. The VR device receives this feedback, generates a slight vibration in the VR user's HMD, and displays an animation expressing happiness. This gives the VR user the feeling of actually being petted, and also allows them to sense the AR user's emotion of happiness.

[0563] Through these steps, the present invention realizes intuitive and emotionally rich interaction that transcends the dimensions of AR and VR.

[0564] Example 2

[0565] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0566] In conventional AR and VR systems, it has been difficult to achieve intuitive and emotion-based interaction across different dimensions. In particular, there has been a lack of systems that can analyze users' gestures and actions in real time and provide appropriate feedback based on that analysis. In addition, there are limited means to accurately recognize users' emotions and generate feedback based on those emotions.

[0567] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0568] In this invention, the server includes a means for transmitting operation information acquired from the AR terminal to the server, a means including a generative AI model for analyzing the operation information and inferring an action, an emotion recognition means for analyzing the operation information and the user's facial expressions and voice and recognizing emotions, a means for generating feedback based on the inferred action and the recognized emotion and transmitting it to the VR terminal, and a means for receiving the feedback and providing a tactile or visual response to the VR user, thereby enabling intuitive and emotional interactions across different dimensions.

[0569] An "AR terminal" is a device that uses augmented reality technology to overlay digital information on the real world.

[0570] "Operation information" is data on the movements and positions of the user's hands and fingers acquired from the AR terminal.

[0571] A "server" is a high-performance computer system that communicates with AR and VR terminals to analyze and process data.

[0572] A "generative AI model" is a model that uses machine learning algorithms to analyze user gesture data and classify and infer specific actions.

[0573] An "emotion recognition means" is a system or algorithm that analyzes the user's voice, facial expressions, gestures, etc. to recognize their emotions.

[0574] "Feedback" refers to the tactile or visual response generated based on actions and perceived emotions.

[0575] A "VR terminal" is a device that uses virtual reality technology to provide an immersive virtual space, and includes a head-mounted display (HMD) and a vibration device.

[0576] "Haptic feedback" refers to a physical stimulus (e.g., vibration) that provides a tactile sensation to the user.

[0577] "Visual response" refers to providing feedback to the user through visual information, including animations and graphics.

[0578] "Interaction across different dimensions" refers to users in the real world and the virtual world sharing and interacting with each other's actions and emotions.

[0579] The system of this invention is composed of an AR terminal, a server, a VR terminal, and emotion recognition means. We will explain a specific embodiment that realizes intuitive and emotional interaction between AR users and VR users across different dimensions through the operation of the entire system.

[0580] Hardware and software used

[0581] AR device: An augmented reality device such as Microsoft HoloLens, equipped with a camera and sensors that captures user movements in real time.

[0582] Server: A high-performance computer system (e.g., AWS EC2 instance) that runs the generative AI model and emotion recognition method.

[0583] VR device: A virtual reality device such as the Oculus Rift, including a head-mounted display (HMD) and a vibration device.

[0584] Emotion recognition method: Uses Microsoft Azure's Emotion API, etc.

[0585] Data processing and calculation

[0586] When a user uses an AR device to make a gesture (e.g., stroking the head) to the VR user's character, the camera and sensors of the AR device capture this gesture and obtain the position information and movement history of the hand and fingers in real time.

[0587] The operation information acquired by the AR device is sent to the server using a real-time communication protocol (e.g., WebSocket). The server receives this data and begins analyzing it using the generative AI model.

[0588] The server uses a generative AI model (for example, a custom model based on TensorFlow) to analyze hand movements and position information and infer actions such as "stroking." Along with the analyzed action data, the server captures the user's facial expressions and voice, and recognizes their emotions using emotion recognition means.

[0589] For example, if the user is smiling and stroking the user's head, the emotion recognition means recognizes the emotion as "joy." The server generates a feedback signal based on the analysis of the user's behavior and emotion.

[0590] The feedback signals include haptic feedback (e.g., vibration of the HMD) and visual feedback (e.g., animation of being petted and graphics expressing pleasure). The server generates the feedback signals and sends them to the VR device.

[0591] The VR device receives the feedback signal sent from the server and provides specific tactile and visual responses to the VR user, such as generating a slight vibration in the HMD and displaying a graphic indicating pleasure along with an animation of the user being stroked.

[0592] Specific examples

[0593] For example, suppose a user uses a tablet to make a gesture of patting a VR user's head while smiling. The AR device captures this gesture and facial expression and sends it to the server. The server's generative AI model analyzes this data to infer the action of "patting," and the emotion recognition means recognizes the emotion of "joy." The server then generates vibration and visual feedback to convey joy and sends it to the VR device. The VR device receives this feedback and generates a slight vibration in the HMD, while simultaneously displaying a graphic that expresses joy along with an animation of the user being stroked. This allows the VR user to experience the sensation of actually being stroked while also sensing the user's emotions.

[0594] Prompt Sentence Examples

[0595] "Imagine an AR user using a tablet to smile and stroke the head of the VR user character through the screen. The AR device captures this gesture and facial expression and sends it to the server in real time. The server's AI model analyzes it and detects the 'stroking' gesture, and the emotion recognition means recognizes the emotion 'happiness'. As a result, the server generates appropriate haptic and visual feedback and sends it to the VR device. The VR device receives this feedback and vibrates the HMD, displaying an animation of the user being stroked and a graphic of happiness."

[0596] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0597] Step 1:

[0598] The user makes a gesture using the AR device. Specifically, the user uses a tablet to pat the head of the VR user's character. At this time, the camera and sensors of the AR device capture the position information and movement history of the hand and fingers in real time. The input is the user's gesture, and the output is real-time position information and movement history.

[0599] Step 2:

[0600] The operation information captured by the device (AR device) is sent to the server via a real-time communication protocol (e.g., WebSocket). Specifically, the AR device forms the position information and movement history of the hands and fingers it has acquired into a data packet and sends it to the server. The input is the real-time position information and movement history, and the output is the sent data packet.

[0601] Step 3:

[0602] The server receives data packets sent from the AR device. Specifically, the server receives the data packets via WebSocket, decodes the packets, and extracts the operation information. The input is the sent data packet, and the output is the operation information.

[0603] Step 4:

[0604] The operation information received by the server is analyzed using a generative AI model. Specifically, the generative AI model (e.g., a custom model based on TensorFlow) analyzes the position information and movement history of the hand and fingers, and infers that the user's gesture is a specific action, such as "stroking." This analysis is performed based on pre-trained gesture classes. The input is operation information, and the output is data on the inferred action.

[0605] Step 5:

[0606] The server uses an emotion recognition means to recognize the user's emotions. Specifically, it analyzes the user's voice, facial expression, and gesture information, and the emotion recognition means (e.g., Microsoft Azure's Emotion API) detects emotions such as "joy." The input is voice, facial expression, and gesture information, and the output is recognized emotion data.

[0607] Step 6:

[0608] The server generates a feedback signal based on the estimated action and the recognized emotion. Specifically, the server combines the action of "stroking" with the emotion of "joy" to generate a signal including haptic feedback (e.g., HMD vibration) and visual feedback (e.g., an animation of being stroked and a graphic expressing joy). The input is action data and emotion data, and the output is a feedback signal.

[0609] Step 7:

[0610] The server generates feedback signals and sends them to the VR device. Specifically, the feedback signals are formed as data packets and sent to the VR device via WebSocket. This provides sensory information to the VR user. The input is the generated feedback signal, and the output is the transmitted data packet.

[0611] Step 8:

[0612] The terminal (VR terminal) receives the feedback signal sent from the server and provides specific tactile and visual responses. Specifically, it generates a slight vibration in the HMD and displays an animation of being stroked and a graphic that expresses pleasure in the VR space. The received feedback signal is the input, and the tactile and visual feedback is provided as the output.

[0613] (Application example 2)

[0614] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0615] It solves the challenge of realizing real-time, emotional interaction between different dimensions, namely augmented reality (AR) and virtual reality (VR), and also improves the customer experience in virtual stores, enabling personalized shopping experiences.

[0616] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means including a generative artificial intelligence model that analyzes operation information acquired from the AR terminal and infers an action, means for generating feedback based on the inferred action and the recognized emotion and transmitting the feedback to the VR terminal, and means for the feedback to include visual and tactile feedback that personalizes the shopping experience in the virtual space. This enables intuitive and emotionally rich interactions between different dimensions and makes it possible to provide a personalized customer experience in the virtual store.

[0617] "Different dimensions" refers to the different realms of existence of augmented reality (AR), which overlays digital information on the real world, and virtual reality (VR), which provides a virtual space.

[0618] A "generative artificial intelligence model" is a modeling method based on machine learning and deep learning that is used to analyze received operational information and infer actions and emotions.

[0619] "Operation information" refers to data such as hand and finger position information and movement history obtained from the AR device.

[0620] "Feedback" refers to a reaction signal generated based on the inferred action and the recognized emotion, to provide the user with a tactile or visual response.

[0621] "Virtual space" refers to a virtual environment or world that can be experienced through a VR device.

[0622] "Gesture data" refers to data related to hand movements and gestures made by a user.

[0623] "Facial expression data" refers to data relating to the user's facial expressions.

[0624] "Promotion information" is information related to sales and special campaigns for specific products.

[0625] "Visual feedback" refers to information provided to a user through their sense of sight, including graphics and animations displayed on a display.

[0626] An "AR terminal" refers to a device that uses augmented reality technology to overlay digital information on the real world.

[0627] A "VR terminal" refers to a device that uses virtual reality technology to provide an immersive virtual space.

[0628] System configuration

[0629] A system for realizing this application example uses the following hardware and software.

[0630] Hardware

[0631] 1. AR device: A device (e.g., smart glasses) that uses augmented reality technology to overlay digital information on the real world.

[0632] 2. VR terminal: A device that uses virtual reality technology to provide an immersive virtual space (e.g., a head-mounted display).

[0633] 3. Server: A high performance computer server.

[0634] software

[0635] 1. Generative AI model: An artificial intelligence model (e.g., TensorFlow or PyTorch) for analyzing hand movements and facial expressions.

[0636] 2. Emotion engine: Software that analyzes voice and facial expression data to recognize emotions (e.g., Google Cloud Natural Language API, Amazon Comprehend).

[0637] 3. Communication protocol: A protocol for communicating data in real time (e.g., WebSocket).

[0638] What the program does

[0639] This system processes data and realizes interactions in the following steps:

[0640] AR device processing

[0641] AR devices are equipped with sensors to capture the user's hand and finger movements and facial expressions. In particular, the AR device acquires the captured gesture and facial expression data in real time and transmits it to a server. For example, if a user points at a product using smart glasses and their facial expression shows delight, that information will be captured.

[0642] Server Processing

[0643] The server is equipped with a generative AI model and emotion engine. The server analyzes the operation information sent from the AR device and estimates the action and emotion. Specifically, it analyzes hand movements and position information to estimate the "pointing" action and further recognizes the emotion of "happiness" from the user's facial expression.

[0644] Generate and send feedback

[0645] Based on the analysis results, the server generates appropriate feedback, including visual and tactile responses. For example, if the estimated action is "pointing" and the emotion is "joy," the server generates visual feedback containing detailed product information and promotional information and sends it to the VR device.

[0646] Applying Feedback

[0647] The VR device responds appropriately in the virtual space based on the received feedback signal, providing visual feedback to the user by displaying product guides or spontaneously displaying promotional information about specific products.

[0648] Examples and prompts

[0649] For example, if a customer uses smart glasses to look at a product in a virtual store, makes a gesture of touching it with their hands, and shows an excited expression, this information is captured and sent to the server. The server analyzes this data and recognizes the behavior of "showing interest in a particular product" and the emotion of "excitement." It then generates feedback on the VR device that displays detailed product information and special discount promotions. The example prompt sentences used by the generative AI model are as follows:

[0650] User is viewing and interacting with a product. Capture the movements and facial expressions, and provide appropriate promotional feedback.

[0651] This makes the user's shopping experience in a virtual store more intuitive and emotionally satisfying.

[0652] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0653] Step 1:

[0654] The AR device captures the user's hand and finger movements and facial expressions in real time. Gesture and facial expression data obtained from sensors is received as input and sent to a server. Specifically, the smart glasses' cameras and sensors capture and measure the user's movements, and the data is processed and compiled as operation information. Operation information includes the position and movement of the hands, facial expressions, etc. This information is sent to the server in real time via WebSocket.

[0655] Step 2:

[0656] The server receives operation information sent from the AR device. The received data includes the user's gestures and facial expression data. The server analyzes this data using a generative AI model and an emotion engine. Specifically, the generative AI model analyzes hand movements and position information to infer actions, and the emotion engine analyzes facial expression data and voice to recognize emotions. As a result of this analysis, the server outputs an action estimation result (e.g., "pointing") and an emotion recognition result (e.g., "excitement").

[0657] Step 3:

[0658] The server generates appropriate feedback based on the estimated action and the recognized emotion. It receives the action estimation results and emotion recognition results as input and generates feedback data accordingly. For example, if the action of "pointing" and the emotion of "excitement" are recognized, the server visually displays detailed information about a product and creates feedback including promotional information related to that product. The generated feedback data is sent to the VR device.

[0659] Step 4:

[0660] The VR device receives the feedback data sent from the server. The received data includes visual feedback and haptic feedback. Specifically, the VR device displays detailed product information and promotional information in the virtual space as visual feedback, and generates device vibrations and other haptic feedback. This process allows the user to experience real-time feedback in the virtual space.

[0661] Step 5:

[0662] Users experience feedback provided through the VR device. Specifically, when a user points at a product in the virtual space, detailed information about that product is displayed. Also, if a user shows interest in a particular product, related promotional information is displayed. This makes the shopping experience in the virtual store more intuitive and emotionally satisfying.

[0663] The above steps enable interaction across different dimensions and improve customer experience in virtual stores.

[0664] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0665] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0666] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0667] [Third embodiment]

[0668] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0669] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0670] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0671] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0672] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0673] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0674] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0675] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0676] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0677] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0678] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0679] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0680] This invention provides a system for realizing interactions between different dimensions. This system consists of an AR terminal, a server, and a VR terminal, and its main processing steps include capturing operation information, transmitting data, analyzing data, generating feedback, and applying feedback.

[0681] System configuration

[0682] AR device

[0683] AR devices are devices that use augmented reality technology to display digital information overlaid on the real world. These devices are equipped with cameras and sensors that capture the position and movement history of the AR user's hands and fingers in real time.

[0684] server

[0685] The server is a high-performance computer system that communicates with the AR and VR devices via a network. The server is equipped with a generative AI model that analyzes the operation information received from the AR device and infers the user's actions.

[0686] VR device

[0687] A VR device is a device that uses virtual reality technology to provide an immersive virtual space. Its main components include a head-mounted display (HMD) and a vibration device. The VR device provides tactile or visual responses to the VR user based on feedback signals sent from the server.

[0688] Program processing and examples

[0689] AR device operation capture

[0690] The AR device captures gestures made by the AR user toward the VR user's character. For example, if the AR user uses a tablet to make a gesture of patting the VR user's character's head through the screen, the sensor will capture the hand movement and position information.

[0691] Sending and Receiving Data

[0692] The AR device transmits the captured operation information to the server, which receives the operation information and stores it in a data queue.

[0693] Data analysis and behavioral inference

[0694] The server uses the generative AI model to analyze the received operation information. For example, it can infer the action of "stroking" based on the hand's trajectory, speed, and movement pattern. The generative AI model then classifies the action using gesture classes it has learned in advance.

[0695] Generate and send feedback

[0696] The server generates feedback based on the analysis results. For example, if it detects a "stroking" action, it generates an appropriate vibration pattern or haptic feedback and sends that signal to the VR device.

[0697] Applying Feedback

[0698] The VR device provides a response to the VR user based on the feedback signal received from the server. Specifically, the VR user's head-mounted device (HMD) generates a slight vibration to recreate the feeling of being stroked. Additionally, an animation of the user being stroked is displayed in the VR space as visual feedback.

[0699] Specific examples

[0700] For example, suppose an AR user uses a tablet to make a gesture of stroking the VR user's head. The AR device captures this gesture and sends it to the server. The server's generative AI model analyzes this data and infers the action of "stroking." The server then generates an appropriate vibration pattern and sends it to the VR device. The VR device receives this feedback and generates a slight vibration in the VR user's HMD while simultaneously displaying an animation of the stroking. This allows the VR user to experience the sensation of actually being stroked.

[0701] This system allows AR and VR users to communicate intuitively and effectively, even though they are in different dimensions.

[0702] The processing flow will be explained below.

[0703] Step 1:

[0704] The AR device captures the gestures made by the AR user. Specifically, the tablet's camera and sensors capture the AR user's hand position (x, y, z coordinates) and movement (velocity, acceleration) in real time. This information is captured frame by frame.

[0705] Step 2:

[0706] The AR device sends the captured operation information to the server, which uses a real-time communication protocol (e.g., WebSocket) to transmit data with low latency.

[0707] Step 3:

[0708] The server receives the operation information sent from the AR device. The received data is temporarily stored in a queue and prepared for analysis by the generative AI model.

[0709] Step 4:

[0710] The server uses the generative AI model to analyze the received operation information. Specifically, it infers the AR user's actions from hand movements and position information. In this process, it uses pre-trained gesture classes to classify actions such as "stroking" and "hitting."

[0711] Step 5:

[0712] The server generates feedback signals based on the analysis results. For example, if the action of "stroking" is detected, it generates appropriate vibration patterns or visual feedback to reproduce the sensation of stroking.

[0713] Step 6:

[0714] The server transmits the generated feedback signals to the VR device, again using a real-time communication protocol to transmit the signals with low latency.

[0715] Step 7:

[0716] The VR device receives the feedback signal sent from the server, which is used as an instruction to provide tactile and visual responses to the VR user.

[0717] Step 8:

[0718] The VR device applies feedback by vibrating the VR user's head-mounted device (HMD) and also provides visual feedback by displaying an animation of the object being stroked in the VR space.

[0719] Step 9:

[0720] The VR user feels the vibrations in their head and visual feedback, allowing them to feel as if the AR user is stroking them, creating a mutual interaction experience that transcends the dimensions of AR and VR.

[0721] Example 1

[0722] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0723] Conventional virtual reality (VR) and augmented reality (AR) systems have the problem of not intuitively reflecting interactions between users in real time. In particular, it is difficult to provide appropriate tactile and visual feedback when multiple users in different dimensions participate in simultaneous communication. Furthermore, accurate recognition of gestures and the quality of feedback can be reduced, resulting in a poor user experience.

[0724] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0725] In this invention, the server includes a means for transmitting operation information acquired from the AR terminal to the server, a means including a generative AI model for analyzing the operation information and inferring an action, and a means for generating feedback based on the inferred action and transmitting the feedback to the VR terminal, thereby enabling intuitive and effective interaction even between users in different dimensions.

[0726] An "AR terminal" is a device that uses augmented reality technology to overlay digital information onto the real world.

[0727] A "server" is a high-performance computer system that communicates with AR and VR terminals via a network.

[0728] "Operation information" is data related to the user's movements and gestures, and is information captured in real time by cameras and sensors.

[0729] A "generative AI model" is a machine learning model that analyzes received operation information and infers user actions, and uses pre-trained gesture classes.

[0730] "Feedback" refers to reaction data generated based on the analysis results, and includes tactile and visual reactions.

[0731] A "VR terminal" is a device that uses virtual reality technology to provide an immersive virtual space, and includes a head-mounted display and a vibration device.

[0732] "Tactile feedback" refers to feedback that provides a direct sensation to the user's body, and includes physical stimuli such as vibrations.

[0733] "Visual response" refers to providing direct feedback to the user's vision, such as on-screen animation.

[0734] A "gesture" is an operation based on the movement of a user's hands or fingers, and is a motion pattern that indicates a specific action.

[0735] A "network" is an infrastructure for realizing communication between a server and a terminal, and includes the Internet and a local area network (LAN).

[0736] A "head-mounted display (HMD)" is a display device worn by the user on the head, providing an immersive VR experience.

[0737] This invention is a system for realizing interactions between different dimensions, and is mainly composed of an AR terminal, a server, and a VR terminal. Each component and its function will be explained in detail below.

[0738] AR device

[0739] AR devices are devices that use augmented reality technology to overlay digital information onto the real world. Specifically, they are equipped with cameras and sensors and have the ability to capture the positional information and movement history of the user's hands and fingers in real time. For example, an AR-enabled tablet can be used, and it can accurately capture the gestures the user makes with a virtual character.

[0740] server

[0741] The server is a high-performance computer system that communicates with the AR and VR devices via a network. It contains a generative AI model that analyzes the operation information sent from the AR device and infers the user's actions. The generative AI model classifies the actions using pre-trained gesture classes and sends the results to the feedback generation module.

[0742] VR device

[0743] A VR device is a device that uses virtual reality technology to provide an immersive virtual space. Its main components are a head-mounted display (HMD) and a vibration device, and it has the function of providing tactile and visual responses to the VR user based on feedback signals sent from the server. Specifically, the VR user's HMD vibrates to reproduce the feeling of being stroked, and displays a stroke animation as visual feedback.

[0744] Specific examples

[0745] For example, if an AR user uses a tablet to make a gesture of stroking the head of a VR user through the screen, the process proceeds as follows: The user operates the AR device, and their hand movements are captured by the camera and sensors. The captured operation information is sent to the server via the network, and the server's generative AI model analyzes the data and infers the action of "stroking." The server then generates an appropriate vibration pattern and visual feedback, which it sends to the VR device. The VR device receives the feedback signal, generates a slight vibration in the HMD, and simultaneously displays an animation of the user being stroked.

[0746] Prompt Sentence Examples

[0747] Specifically, the following prompts are used:

[0748] "Analyze the following operation information to generate feedback as if the VR user's character is being stroked: hand movement trajectory, speed, and movement pattern: {hand movement data}"

[0749] This system enables AR and VR users to communicate intuitively and effectively in real time, even though they are in different dimensions.

[0750] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0751] Step 1:

[0752] Operation capture using an AR device

[0753] Input: User gestures

[0754] Output: Hand position information, movement history

[0755] Specific behavior:

[0756] The user operates the AR device and touches a virtual object (e.g., the head of a VR character) through the screen. The camera and sensors on the AR device capture this action in real time. Specifically, the position information and movement history of the hands and fingers are acquired and converted into data.

[0757] Step 2:

[0758] Sending data and receiving it on the server

[0759] Input: Hand position information, movement history

[0760] Output: Sending operation information data, storing received data

[0761] Specific behavior:

[0762] The AR device sends the hand position information and movement history to the server. The data is transferred via the network, and the server receives the data and temporarily stores it in a data queue.

[0763] Step 3:

[0764] Data analysis and behavioral inference

[0765] Input: Operation information data

[0766] Output: Estimated result of the action

[0767] Specific behavior:

[0768] The server uses the generated AI model to analyze the received operation information data. Specifically, it analyzes the trajectory, speed, and movement patterns of the hand movement to infer actions such as "stroking." For example, from an analysis of the smooth movement of the hand and its speed, it can infer with a high degree of probability that the action is "stroking."

[0769] Step 4:

[0770] Generate and send feedback

[0771] Input: Estimated result of the action

[0772] Output: Feedback data

[0773] Specific behavior:

[0774] The server generates appropriate feedback based on the action estimation. For example, a vibration pattern or visual feedback (animation) corresponding to the action of "stroking" is generated. The generated feedback data is sent to the VR device.

[0775] Step 5:

[0776] Applying Feedback

[0777] Input: Feedback data

[0778] Output: Tactile and visual feedback to the user

[0779] Specific behavior:

[0780] The VR device responds to the user based on the feedback data received from the server. The HMD generates appropriate vibrations and displays animations of the character being stroked in the VR space. For example, the visual effect of a character's head being stroked along with a slight vibration gives the user the sensation of actually stroking the character.

[0781] (Application example 1)

[0782] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0783] While the direct interaction between customers and store staff forms part of the value of the shopping experience in a physical store, there are limitations in that not all customers can receive support at the same time. Another issue is that when store staff provide support remotely, the information and experience customers receive is limited. This can lead to customers not being able to fully understand how to use a product or its features.

[0784] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0785] In this invention, the server includes: means for transmitting operation information acquired from the AR device to the server; means including a generative AI model that analyzes the operation information and infers actions; means for generating feedback based on the inferred actions and transmitting it to the VR device; means for receiving the feedback and providing tactile or visual feedback to the VR user; means for a customer to wear AR glasses in a physical store and for a remote store staff member to provide product operation instructions using VR gloves; and means for the server to analyze the customer's behavior, generate appropriate feedback, and display it on the AR glasses. This allows customers to intuitively and effectively understand how to use and use products while providing valuable remote support.

[0786] "Interaction between different dimensions" refers to the interaction and information exchange between users that takes place between a virtual space, which is different from the physical space, and a real space using virtual reality or augmented reality.

[0787] An "AR terminal" is a device that uses augmented reality technology to overlay digital information on the real world, and is equipped with a camera and sensors.

[0788] "Operation information" refers to data acquired by the AR device when it recognizes operations such as user movements and gestures.

[0789] A "server" is a high-performance computer system that communicates with various terminals via a network and analyzes data.

[0790] A "generative AI model" is an algorithm or model that uses artificial intelligence to analyze data and infer user operations and behavior.

[0791] A "VR terminal" is a device that uses virtual reality technology to provide users with an immersive virtual space, and includes head-mounted displays and haptic devices.

[0792] "Feedback" refers to the tactile or visual response that is generated in response to a user's operation or action.

[0793] A "brick and mortar store" is a commercial establishment that exists in a physical location and can be visited in person by customers.

[0794] "AR glasses" are glasses-type devices that use augmented reality technology, allowing users to visually view digital information overlaid on the real world.

[0795] "Remote store staff" refers to store staff who are physically located in a different location from the physical store but who support customers virtually.

[0796] "VR gloves" are glove-shaped devices that allow users to operate the device with their hands in a virtual reality world and provide haptic feedback.

[0797] "Analyzing customer behavior" means collecting data on the operations and actions performed by customers, and analyzing that data to infer specific actions.

[0798] This invention provides a system that uses an AR terminal, a server, and a VR terminal to improve the shopping experience in a physical store. This system allows customers to wear AR glasses in the physical store, and remote store staff can use VR gloves to provide instructions on how to operate products.

[0799] First, a customer wears AR glasses and walks around a physical store. The AR device (e.g., a glasses-type device that uses augmented reality technology) captures customer operation information through cameras and sensors. For example, when a customer looks at a product or makes a gesture of reaching out, that operation information is acquired in real time.

[0800] This operation information is then sent to a server and stored in a data queue. The server is a high-performance computer system (e.g., an AWS EC2 instance) that analyzes the received operation information using a generative AI model (e.g., OpenAI's GPT-4). Specifically, the server analyzes hand movements, gaze direction, and movement patterns to infer the action of "selecting a product."

[0801] Based on the analyzed actions, the server generates feedback. A remote store staff member wearing VR gloves (e.g., a glove-type device for operating with hands in a virtual reality world) can use the feedback to demonstrate to the customer how to operate the product and its characteristics. The server generates appropriate feedback (visual and tactile responses) and displays it on the AR glasses, allowing the customer to experience the store staff as if they were right in front of them.

[0802] Here is one concrete example:

[0803] 1. A customer puts on AR glasses in a physical store and looks at a specific product.

[0804] 2. The AR device captures the eye movements and hand gestures and sends the operation information to the server.

[0805] 3. The server's generated AI model analyzes the operation information and infers the action of "selecting a product."

[0806] 4. The server sends notifications to remote store staff based on the analysis results.

[0807] 5. Remote store staff will use VR gloves to virtually demonstrate how to operate the product.

[0808] 6. Feedback is displayed on the customer's AR glasses, allowing them to visually confirm how to use the product and its features.

[0809] Example prompts for generative AI models:

[0810] Analyze the following input data and infer the actions taken by the customer (product selection, inquiry, etc.).

[0811] Input data: { "Action": "Reach out", "Target": "Product code 12345", "Timestamp": "2023-10-01T08:00:00Z"}

[0812] Please output the analysis results in the following format:

[0813] {

[0814] "Action": "Product Selection",

[0815] "Product Code": "12345",

[0816] "Recommended feedback": "View a demonstration of how to use the product"

[0817] }

[0818] The invention makes the in-store shopping experience more intuitive and effective, allowing customers to understand the usage and characteristics of products with real-time support from remote store staff.

[0819] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0820] Step 1:

[0821] The AR device (AR glasses) captures customer operation information. Specifically, it uses cameras and sensors to detect the customer's gaze and hand movements and collects that data. The input is the customer's gaze direction and hand gestures, and the output is data that compiles this operation information.

[0822] Step 2:

[0823] The AR device sends the captured operation information to the server. Specifically, the operation information is transferred to the server in real time via the network. The input is the operation information data obtained in step 1, and the output is the operation information data received by the server.

[0824] Step 3:

[0825] The server stores the received operation information in a data queue. Specifically, the server temporarily saves the operation information and prepares it for analysis. The input is the operation information data sent from the AR terminal, and the output is the operation information stored in the data queue.

[0826] Step 4:

[0827] The server's generative AI model analyzes the operation information. Specifically, the operation information is input into a generative AI model (e.g., GPT-4) to infer the customer's actions. In this process, hand movements and gaze patterns are recognized to infer specific actions (e.g., product selection). The input is the operation information stored in the data queue, and the output is the inferred action result.

[0828] Step 5:

[0829] The server generates feedback based on the inference results. Specifically, it generates feedback data (visual and tactile responses) according to the inferred actions and sends a notification to the remote store staff. The input is the inference results of the generative AI model, and the output is feedback data sent to the remote store staff and the VR gloves.

[0830] Step 6:

[0831] Remote store staff use VR gloves to virtually demonstrate to customers how to operate products. Specifically, they operate the products through the VR gloves and send the operation information to the server. The input is a feedback notification from the server, and the output is feedback data as a demonstration of the operation.

[0832] Step 7:

[0833] The server analyzes the operation information of the remote store staff and generates feedback. Specifically, it analyzes the operation information of the remote store staff and generates visual and tactile feedback to be displayed on the AR terminal (AR glasses). The input is the operation information from the remote store staff, and the output is feedback data sent to the AR glasses.

[0834] Step 8:

[0835] The AR device (AR glasses) provides the customer with the feedback received from the server. Specifically, it displays guides and arrows that visually show how to use the product, and provides tactile feedback as needed. The input is the feedback data from the server, and the output is the visual and tactile response that the customer actually experiences.

[0836] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0837] This invention provides a system for realizing interactions across different dimensions. This system consists of an AR terminal, a server, a VR terminal, and an emotion engine, and its main processing steps include capturing operation information, transmitting data, analyzing data, recognizing emotions, generating feedback, and applying feedback.

[0838] System configuration

[0839] AR device

[0840] AR devices are devices that use augmented reality technology to display digital information overlaid on the real world. These devices are equipped with cameras and sensors that capture the position and movement history of the AR user's hands and fingers in real time.

[0841] server

[0842] The server is a high-performance computer system that communicates with the AR and VR devices via a network. The server is equipped with a generative AI model and an emotion engine, and analyzes the operation information received from the AR device to estimate the user's actions and emotions.

[0843] VR device

[0844] A VR device is a device that uses virtual reality technology to provide an immersive virtual space. Its main components include a head-mounted display (HMD) and a vibration device. The VR device provides tactile or visual responses to the VR user based on feedback signals sent from the server.

[0845] Emotion Engine

[0846] The emotion engine is a software component that recognizes the emotions of AR users from their voices, facial expressions, gestures, etc. This allows it to generate more accurate feedback by combining it with operational information.

[0847] Program processing and examples

[0848] AR device operation capture

[0849] The AR device captures gestures made by the AR user toward the VR user's character. For example, if the AR user uses a tablet to make a gesture of patting the VR user's character's head through the screen, the sensor will capture the hand movement and position information.

[0850] Sending and Receiving Data

[0851] The AR device sends the captured operation information to the server, using a real-time communication protocol (e.g., WebSocket) to transmit data with low latency.

[0852] Data analysis and behavioral inference

[0853] The server uses the generated AI model to analyze the received operation information. Specifically, it infers the AR user's actions from hand movements and position information. In this process, it uses pre-trained gesture classes to classify actions such as "stroking" and "hitting."

[0854] emotion recognition

[0855] The server's emotion engine analyzes the AR user's voice, facial expressions, and gestures to recognize their emotions. For example, if the AR user is smiling and patting someone on the head, the emotion of "joy" is recognized.

[0856] Generate and send feedback

[0857] The server generates a feedback signal based on the analysis results and the recognized emotion. For example, if the action of "stroking" and the emotion of "joy" are inferred, the server generates an appropriate vibration pattern and visual feedback that conveys joy, and sends that signal to the VR device.

[0858] Applying Feedback

[0859] The VR device receives the feedback signal sent from the server. The received signal is used as an instruction to provide tactile and visual responses to the VR user. Specifically, the VR user's head-mounted device (HMD) generates vibrations, and animations of being petted or graphics conveying pleasure are displayed in the VR space.

[0860] Specific examples

[0861] For example, suppose an AR user uses a tablet to make a gesture of patting a VR user's head while smiling. The AR device captures this gesture and facial expression and sends it to the server. The server's generative AI model analyzes this data to infer the action of "patting," and the emotion engine recognizes the emotion of "joy." The server then generates vibration and visual feedback to convey joy and sends it to the VR device. The VR device receives this feedback and generates a slight vibration in the VR user's HMD, while simultaneously displaying a graphic that expresses joy along with an animation of the pat. This allows the VR user to experience the sensation of actually being petted while also sensing the AR user's emotions.

[0862] This system allows AR and VR users to interact intuitively and emotionally, even though they are in different dimensions.

[0863] The processing flow will be explained below.

[0864] Step 1:

[0865] The AR device captures the gestures made by the AR user. Specifically, the tablet's camera and sensors capture the AR user's hand position (x, y, z coordinates) and movement (velocity, acceleration) in real time. This information is captured frame by frame.

[0866] Step 2:

[0867] The AR device sends the acquired gesture data to the server using a real-time communication protocol (e.g., WebSocket) to transmit the data with low latency.

[0868] Step 3:

[0869] The server receives the gesture data sent from the AR device and temporarily stores the received data in a queue.

[0870] Step 4:

[0871] The server uses an emotion engine to recognize the emotions of the AR user. For example, the server analyzes the voice and facial expression data received from the AR device to identify emotions such as "joy" or "sadness."

[0872] Step 5:

[0873] The server analyzes the received gesture data using a generative AI model to infer the AR user's actions. For example, it can infer the action of "stroking" from hand movements and position information.

[0874] Step 6:

[0875] The server generates a feedback signal based on the analysis results and the recognized emotion. For example, if the action of "stroking" and the emotion of "joy" are inferred, the server generates a specific vibration pattern and visual feedback that conveys joy.

[0876] Step 7:

[0877] The server transmits the generated feedback signal to the VR device using a real-time communication protocol with low latency.

[0878] Step 8:

[0879] The VR device receives the feedback signals sent from the server, which are used to provide haptic and visual responses.

[0880] Step 9:

[0881] The VR device applies a reaction to the VR user based on the received feedback signal. Specifically, it generates vibrations in the VR user's HMD and displays animations of being petted and graphics that convey pleasure in the VR space.

[0882] Step 10:

[0883] The VR user feels the AR user stroking them through vibrations in the head and visual feedback, and can also sense the AR user's emotions based on the feedback.

[0884] As a specific example, consider a scene in which an AR user uses a tablet to make a gesture of patting a VR user's head, smiling while doing so. The AR device captures the gesture and facial expression and sends the data to the server. The server's emotion engine recognizes the smile as the emotion of "happiness," and the generative AI model infers the action of "patting." The server then generates a feedback signal based on the emotion and action and sends it to the VR device. The VR device receives this feedback, generates a slight vibration in the VR user's HMD, and displays an animation expressing happiness. This gives the VR user the feeling of actually being petted, and also allows them to sense the AR user's emotion of happiness.

[0885] Through these steps, the present invention realizes intuitive and emotionally rich interaction that transcends the dimensions of AR and VR.

[0886] Example 2

[0887] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0888] In conventional AR and VR systems, it has been difficult to achieve intuitive and emotion-based interaction across different dimensions. In particular, there has been a lack of systems that can analyze users' gestures and actions in real time and provide appropriate feedback based on that analysis. In addition, there are limited means to accurately recognize users' emotions and generate feedback based on those emotions.

[0889] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0890] In this invention, the server includes a means for transmitting operation information acquired from the AR terminal to the server, a means including a generative AI model for analyzing the operation information and inferring an action, an emotion recognition means for analyzing the operation information and the user's facial expressions and voice and recognizing emotions, a means for generating feedback based on the inferred action and the recognized emotion and transmitting it to the VR terminal, and a means for receiving the feedback and providing a tactile or visual response to the VR user, thereby enabling intuitive and emotional interactions across different dimensions.

[0891] An "AR terminal" is a device that uses augmented reality technology to overlay digital information on the real world.

[0892] "Operation information" is data on the movements and positions of the user's hands and fingers acquired from the AR terminal.

[0893] A "server" is a high-performance computer system that communicates with AR and VR terminals to analyze and process data.

[0894] A "generative AI model" is a model that uses machine learning algorithms to analyze user gesture data and classify and infer specific actions.

[0895] An "emotion recognition means" is a system or algorithm that analyzes the user's voice, facial expressions, gestures, etc. to recognize their emotions.

[0896] "Feedback" refers to the tactile or visual response generated based on actions and perceived emotions.

[0897] A "VR terminal" is a device that uses virtual reality technology to provide an immersive virtual space, and includes a head-mounted display (HMD) and a vibration device.

[0898] "Haptic feedback" refers to a physical stimulus (e.g., vibration) that provides a tactile sensation to the user.

[0899] "Visual response" refers to providing feedback to the user through visual information, including animations and graphics.

[0900] "Interaction across different dimensions" refers to users in the real world and the virtual world sharing and interacting with each other's actions and emotions.

[0901] The system of this invention is composed of an AR terminal, a server, a VR terminal, and emotion recognition means. We will explain a specific embodiment that realizes intuitive and emotional interaction between AR users and VR users across different dimensions through the operation of the entire system.

[0902] Hardware and software used

[0903] AR device: An augmented reality device such as Microsoft HoloLens, equipped with a camera and sensors that captures user movements in real time.

[0904] Server: A high-performance computer system (e.g., AWS EC2 instance) that runs the generative AI model and emotion recognition method.

[0905] VR device: A virtual reality device such as the Oculus Rift, including a head-mounted display (HMD) and a vibration device.

[0906] Emotion recognition method: Uses Microsoft Azure's Emotion API, etc.

[0907] Data processing and calculation

[0908] When a user uses an AR device to make a gesture (e.g., stroking the head) to the VR user's character, the camera and sensors of the AR device capture this gesture and obtain the position information and movement history of the hand and fingers in real time.

[0909] The operation information acquired by the AR device is sent to the server using a real-time communication protocol (e.g., WebSocket). The server receives this data and begins analyzing it using the generative AI model.

[0910] The server uses a generative AI model (for example, a custom model based on TensorFlow) to analyze hand movements and position information and infer actions such as "stroking." Along with the analyzed action data, the server captures the user's facial expressions and voice, and recognizes their emotions using emotion recognition means.

[0911] For example, if the user is smiling and stroking the user's head, the emotion recognition means recognizes the emotion as "joy." The server generates a feedback signal based on the analysis of the user's behavior and emotion.

[0912] The feedback signals include haptic feedback (e.g., vibration of the HMD) and visual feedback (e.g., animation of being petted and graphics expressing pleasure). The server generates the feedback signals and sends them to the VR device.

[0913] The VR device receives the feedback signal sent from the server and provides specific tactile and visual responses to the VR user, such as generating a slight vibration in the HMD and displaying a graphic indicating pleasure along with an animation of the user being stroked.

[0914] Specific examples

[0915] For example, suppose a user uses a tablet to make a gesture of patting a VR user's head while smiling. The AR device captures this gesture and facial expression and sends it to the server. The server's generative AI model analyzes this data to infer the action of "patting," and the emotion recognition means recognizes the emotion of "joy." The server then generates vibration and visual feedback to convey joy and sends it to the VR device. The VR device receives this feedback and generates a slight vibration in the HMD, while simultaneously displaying a graphic that expresses joy along with an animation of the user being stroked. This allows the VR user to experience the sensation of actually being stroked while also sensing the user's emotions.

[0916] Prompt Sentence Examples

[0917] "Imagine an AR user using a tablet to smile and stroke the head of the VR user character through the screen. The AR device captures this gesture and facial expression and sends it to the server in real time. The server's AI model analyzes it and detects the 'stroking' gesture, and the emotion recognition means recognizes the emotion 'happiness'. As a result, the server generates appropriate haptic and visual feedback and sends it to the VR device. The VR device receives this feedback and vibrates the HMD, displaying an animation of the user being stroked and a graphic of happiness."

[0918] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0919] Step 1:

[0920] The user makes a gesture using the AR device. Specifically, the user uses a tablet to pat the head of the VR user's character. At this time, the camera and sensors of the AR device capture the position information and movement history of the hand and fingers in real time. The input is the user's gesture, and the output is real-time position information and movement history.

[0921] Step 2:

[0922] The operation information captured by the device (AR device) is sent to the server via a real-time communication protocol (e.g., WebSocket). Specifically, the AR device forms the position information and movement history of the hands and fingers it has acquired into a data packet and sends it to the server. The input is the real-time position information and movement history, and the output is the sent data packet.

[0923] Step 3:

[0924] The server receives data packets sent from the AR device. Specifically, the server receives the data packets via WebSocket, decodes the packets, and extracts the operation information. The input is the sent data packet, and the output is the operation information.

[0925] Step 4:

[0926] The operation information received by the server is analyzed using a generative AI model. Specifically, the generative AI model (e.g., a custom model based on TensorFlow) analyzes the position information and movement history of the hand and fingers, and infers that the user's gesture is a specific action, such as "stroking." This analysis is performed based on pre-trained gesture classes. The input is operation information, and the output is data on the inferred action.

[0927] Step 5:

[0928] The server uses an emotion recognition means to recognize the user's emotions. Specifically, it analyzes the user's voice, facial expression, and gesture information, and the emotion recognition means (e.g., Microsoft Azure's Emotion API) detects emotions such as "joy." The input is voice, facial expression, and gesture information, and the output is recognized emotion data.

[0929] Step 6:

[0930] The server generates a feedback signal based on the estimated action and the recognized emotion. Specifically, the server combines the action of "stroking" with the emotion of "joy" to generate a signal including haptic feedback (e.g., HMD vibration) and visual feedback (e.g., an animation of being stroked and a graphic expressing joy). The input is action data and emotion data, and the output is a feedback signal.

[0931] Step 7:

[0932] The server generates feedback signals and sends them to the VR device. Specifically, the feedback signals are formed as data packets and sent to the VR device via WebSocket. This provides sensory information to the VR user. The input is the generated feedback signal, and the output is the transmitted data packet.

[0933] Step 8:

[0934] The terminal (VR terminal) receives the feedback signal sent from the server and provides specific tactile and visual responses. Specifically, it generates a slight vibration in the HMD and displays an animation of being stroked and a graphic that expresses pleasure in the VR space. The received feedback signal is the input, and the tactile and visual feedback is provided as the output.

[0935] (Application example 2)

[0936] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0937] It solves the challenge of realizing real-time, emotional interaction between different dimensions, namely augmented reality (AR) and virtual reality (VR), and also improves the customer experience in virtual stores, enabling personalized shopping experiences.

[0938] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means including a generative artificial intelligence model that analyzes operation information acquired from the AR terminal and infers an action, means for generating feedback based on the inferred action and the recognized emotion and transmitting the feedback to the VR terminal, and means for the feedback to include visual and tactile feedback that personalizes the shopping experience in the virtual space. This enables intuitive and emotionally rich interactions between different dimensions and makes it possible to provide a personalized customer experience in the virtual store.

[0939] "Different dimensions" refers to the different realms of existence of augmented reality (AR), which overlays digital information on the real world, and virtual reality (VR), which provides a virtual space.

[0940] A "generative artificial intelligence model" is a modeling method based on machine learning and deep learning that is used to analyze received operational information and infer actions and emotions.

[0941] "Operation information" refers to data such as hand and finger position information and movement history obtained from the AR device.

[0942] "Feedback" refers to a reaction signal generated based on the inferred action and the recognized emotion, to provide the user with a tactile or visual response.

[0943] "Virtual space" refers to a virtual environment or world that can be experienced through a VR device.

[0944] "Gesture data" refers to data related to hand movements and gestures made by a user.

[0945] "Facial expression data" refers to data relating to the user's facial expressions.

[0946] "Promotion information" is information related to sales and special campaigns for specific products.

[0947] "Visual feedback" refers to information provided to a user through their sense of sight, including graphics and animations displayed on a display.

[0948] An "AR terminal" refers to a device that uses augmented reality technology to overlay digital information on the real world.

[0949] A "VR terminal" refers to a device that uses virtual reality technology to provide an immersive virtual space.

[0950] System configuration

[0951] A system for realizing this application example uses the following hardware and software.

[0952] Hardware

[0953] 1. AR device: A device (e.g., smart glasses) that uses augmented reality technology to overlay digital information on the real world.

[0954] 2. VR terminal: A device that uses virtual reality technology to provide an immersive virtual space (e.g., a head-mounted display).

[0955] 3. Server: A high performance computer server.

[0956] software

[0957] 1. Generative AI model: An artificial intelligence model (e.g., TensorFlow or PyTorch) for analyzing hand movements and facial expressions.

[0958] 2. Emotion engine: Software that analyzes voice and facial expression data to recognize emotions (e.g., Google Cloud Natural Language API, Amazon Comprehend).

[0959] 3. Communication protocol: A protocol for communicating data in real time (e.g., WebSocket).

[0960] What the program does

[0961] This system processes data and realizes interactions in the following steps:

[0962] AR device processing

[0963] AR devices are equipped with sensors to capture the user's hand and finger movements and facial expressions. In particular, the AR device acquires the captured gesture and facial expression data in real time and transmits it to a server. For example, if a user points at a product using smart glasses and their facial expression shows delight, that information will be captured.

[0964] Server Processing

[0965] The server is equipped with a generative AI model and emotion engine. The server analyzes the operation information sent from the AR device and estimates the action and emotion. Specifically, it analyzes hand movements and position information to estimate the "pointing" action and further recognizes the emotion of "happiness" from the user's facial expression.

[0966] Generate and send feedback

[0967] Based on the analysis results, the server generates appropriate feedback, including visual and tactile responses. For example, if the estimated action is "pointing" and the emotion is "joy," the server generates visual feedback containing detailed product information and promotional information and sends it to the VR device.

[0968] Applying Feedback

[0969] The VR device responds appropriately in the virtual space based on the received feedback signal, providing visual feedback to the user by displaying product guides or spontaneously displaying promotional information about specific products.

[0970] Examples and prompts

[0971] For example, if a customer uses smart glasses to look at a product in a virtual store, makes a gesture of touching it with their hands, and shows an excited expression, this information is captured and sent to the server. The server analyzes this data and recognizes the behavior of "showing interest in a particular product" and the emotion of "excitement." It then generates feedback on the VR device that displays detailed product information and special discount promotions. The example prompt sentences used by the generative AI model are as follows:

[0972] User is viewing and interacting with a product. Capture the movements and facial expressions, and provide appropriate promotional feedback.

[0973] This makes the user's shopping experience in a virtual store more intuitive and emotionally satisfying.

[0974] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0975] Step 1:

[0976] The AR device captures the user's hand and finger movements and facial expressions in real time. Gesture and facial expression data obtained from sensors is received as input and sent to a server. Specifically, the smart glasses' cameras and sensors capture and measure the user's movements, and the data is processed and compiled as operation information. Operation information includes the position and movement of the hands, facial expressions, etc. This information is sent to the server in real time via WebSocket.

[0977] Step 2:

[0978] The server receives operation information sent from the AR device. The received data includes the user's gestures and facial expression data. The server analyzes this data using a generative AI model and an emotion engine. Specifically, the generative AI model analyzes hand movements and position information to infer actions, and the emotion engine analyzes facial expression data and voice to recognize emotions. As a result of this analysis, the server outputs an action estimation result (e.g., "pointing") and an emotion recognition result (e.g., "excitement").

[0979] Step 3:

[0980] The server generates appropriate feedback based on the estimated action and the recognized emotion. It receives the action estimation results and emotion recognition results as input and generates feedback data accordingly. For example, if the action of "pointing" and the emotion of "excitement" are recognized, the server visually displays detailed information about a product and creates feedback including promotional information related to that product. The generated feedback data is sent to the VR device.

[0981] Step 4:

[0982] The VR device receives the feedback data sent from the server. The received data includes visual feedback and haptic feedback. Specifically, the VR device displays detailed product information and promotional information in the virtual space as visual feedback, and generates device vibrations and other haptic feedback. This process allows the user to experience real-time feedback in the virtual space.

[0983] Step 5:

[0984] Users experience feedback provided through the VR device. Specifically, when a user points at a product in the virtual space, detailed information about that product is displayed. Also, if a user shows interest in a particular product, related promotional information is displayed. This makes the shopping experience in the virtual store more intuitive and emotionally satisfying.

[0985] The above steps enable interaction across different dimensions and improve customer experience in virtual stores.

[0986] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0987] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0988] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0989] [Fourth embodiment]

[0990] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0991] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0992] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0993] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0994] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0995] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0996] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0997] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0998] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0999] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1000] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1001] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1002] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1003] This invention provides a system for realizing interactions between different dimensions. This system consists of an AR terminal, a server, and a VR terminal, and its main processing steps include capturing operation information, transmitting data, analyzing data, generating feedback, and applying feedback.

[1004] System configuration

[1005] AR device

[1006] AR devices are devices that use augmented reality technology to display digital information overlaid on the real world. These devices are equipped with cameras and sensors that capture the position and movement history of the AR user's hands and fingers in real time.

[1007] server

[1008] The server is a high-performance computer system that communicates with the AR and VR devices via a network. The server is equipped with a generative AI model that analyzes the operation information received from the AR device and infers the user's actions.

[1009] VR device

[1010] A VR device is a device that uses virtual reality technology to provide an immersive virtual space. Its main components include a head-mounted display (HMD) and a vibration device. The VR device provides tactile or visual responses to the VR user based on feedback signals sent from the server.

[1011] Program processing and examples

[1012] AR device operation capture

[1013] The AR device captures gestures made by the AR user toward the VR user's character. For example, if the AR user uses a tablet to make a gesture of patting the VR user's character's head through the screen, the sensor will capture the hand movement and position information.

[1014] Sending and Receiving Data

[1015] The AR device transmits the captured operation information to the server, which receives the operation information and stores it in a data queue.

[1016] Data analysis and behavioral inference

[1017] The server uses the generative AI model to analyze the received operation information. For example, it can infer the action of "stroking" based on the hand's trajectory, speed, and movement pattern. The generative AI model then classifies the action using gesture classes it has learned in advance.

[1018] Generate and send feedback

[1019] The server generates feedback based on the analysis results. For example, if it detects a "stroking" action, it generates an appropriate vibration pattern or haptic feedback and sends that signal to the VR device.

[1020] Applying Feedback

[1021] The VR device provides a response to the VR user based on the feedback signal received from the server. Specifically, the VR user's head-mounted device (HMD) generates a slight vibration to recreate the feeling of being stroked. Additionally, an animation of the user being stroked is displayed in the VR space as visual feedback.

[1022] Specific examples

[1023] For example, suppose an AR user uses a tablet to make a gesture of stroking the VR user's head. The AR device captures this gesture and sends it to the server. The server's generative AI model analyzes this data and infers the action of "stroking." The server then generates an appropriate vibration pattern and sends it to the VR device. The VR device receives this feedback and generates a slight vibration in the VR user's HMD while simultaneously displaying an animation of the stroking. This allows the VR user to experience the sensation of actually being stroked.

[1024] This system allows AR and VR users to communicate intuitively and effectively, even though they are in different dimensions.

[1025] The processing flow will be explained below.

[1026] Step 1:

[1027] The AR device captures the gestures made by the AR user. Specifically, the tablet's camera and sensors capture the AR user's hand position (x, y, z coordinates) and movement (velocity, acceleration) in real time. This information is captured frame by frame.

[1028] Step 2:

[1029] The AR device sends the captured operation information to the server, which uses a real-time communication protocol (e.g., WebSocket) to transmit data with low latency.

[1030] Step 3:

[1031] The server receives the operation information sent from the AR device. The received data is temporarily stored in a queue and prepared for analysis by the generative AI model.

[1032] Step 4:

[1033] The server uses the generative AI model to analyze the received operation information. Specifically, it infers the AR user's actions from hand movements and position information. In this process, it uses pre-trained gesture classes to classify actions such as "stroking" and "hitting."

[1034] Step 5:

[1035] The server generates feedback signals based on the analysis results. For example, if the action of "stroking" is detected, it generates appropriate vibration patterns or visual feedback to reproduce the sensation of stroking.

[1036] Step 6:

[1037] The server transmits the generated feedback signals to the VR device, again using a real-time communication protocol to transmit the signals with low latency.

[1038] Step 7:

[1039] The VR device receives the feedback signal sent from the server, which is used as an instruction to provide tactile and visual responses to the VR user.

[1040] Step 8:

[1041] The VR device applies feedback by vibrating the VR user's head-mounted device (HMD) and also provides visual feedback by displaying an animation of the object being stroked in the VR space.

[1042] Step 9:

[1043] The VR user feels the vibrations in their head and visual feedback, allowing them to feel as if the AR user is stroking them, creating a mutual interaction experience that transcends the dimensions of AR and VR.

[1044] Example 1

[1045] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1046] Conventional virtual reality (VR) and augmented reality (AR) systems have the problem of not intuitively reflecting interactions between users in real time. In particular, it is difficult to provide appropriate tactile and visual feedback when multiple users in different dimensions participate in simultaneous communication. Furthermore, accurate recognition of gestures and the quality of feedback can be reduced, resulting in a poor user experience.

[1047] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1048] In this invention, the server includes a means for transmitting operation information acquired from the AR terminal to the server, a means including a generative AI model for analyzing the operation information and inferring an action, and a means for generating feedback based on the inferred action and transmitting the feedback to the VR terminal, thereby enabling intuitive and effective interaction even between users in different dimensions.

[1049] An "AR terminal" is a device that uses augmented reality technology to overlay digital information onto the real world.

[1050] A "server" is a high-performance computer system that communicates with AR and VR terminals via a network.

[1051] "Operation information" is data related to the user's movements and gestures, and is information captured in real time by cameras and sensors.

[1052] A "generative AI model" is a machine learning model that analyzes received operation information and infers user actions, and uses pre-trained gesture classes.

[1053] "Feedback" refers to reaction data generated based on the analysis results, and includes tactile and visual reactions.

[1054] A "VR terminal" is a device that uses virtual reality technology to provide an immersive virtual space, and includes a head-mounted display and a vibration device.

[1055] "Tactile feedback" refers to feedback that provides a direct sensation to the user's body, and includes physical stimuli such as vibrations.

[1056] "Visual response" refers to providing direct feedback to the user's vision, such as on-screen animation.

[1057] A "gesture" is an operation based on the movement of a user's hands or fingers, and is a motion pattern that indicates a specific action.

[1058] A "network" is an infrastructure for realizing communication between a server and a terminal, and includes the Internet and a local area network (LAN).

[1059] A "head-mounted display (HMD)" is a display device worn by the user on the head, providing an immersive VR experience.

[1060] This invention is a system for realizing interactions between different dimensions, and is mainly composed of an AR terminal, a server, and a VR terminal. Each component and its function will be explained in detail below.

[1061] AR device

[1062] AR devices are devices that use augmented reality technology to overlay digital information onto the real world. Specifically, they are equipped with cameras and sensors and have the ability to capture the positional information and movement history of the user's hands and fingers in real time. For example, an AR-enabled tablet can be used, and it can accurately capture the gestures the user makes with a virtual character.

[1063] server

[1064] The server is a high-performance computer system that communicates with the AR and VR devices via a network. It contains a generative AI model that analyzes the operation information sent from the AR device and infers the user's actions. The generative AI model classifies the actions using pre-trained gesture classes and sends the results to the feedback generation module.

[1065] VR device

[1066] A VR device is a device that uses virtual reality technology to provide an immersive virtual space. Its main components are a head-mounted display (HMD) and a vibration device, and it has the function of providing tactile and visual responses to the VR user based on feedback signals sent from the server. Specifically, the VR user's HMD vibrates to reproduce the feeling of being stroked, and displays a stroke animation as visual feedback.

[1067] Specific examples

[1068] For example, if an AR user uses a tablet to make a gesture of stroking the head of a VR user through the screen, the process proceeds as follows: The user operates the AR device, and their hand movements are captured by the camera and sensors. The captured operation information is sent to the server via the network, and the server's generative AI model analyzes the data and infers the action of "stroking." The server then generates an appropriate vibration pattern and visual feedback, which it sends to the VR device. The VR device receives the feedback signal, generates a slight vibration in the HMD, and simultaneously displays an animation of the user being stroked.

[1069] Prompt Sentence Examples

[1070] Specifically, the following prompts are used:

[1071] "Analyze the following operation information to generate feedback as if the VR user's character is being stroked: hand movement trajectory, speed, and movement pattern: {hand movement data}"

[1072] This system enables AR and VR users to communicate intuitively and effectively in real time, even though they are in different dimensions.

[1073] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1074] Step 1:

[1075] Operation capture using an AR device

[1076] Input: User gestures

[1077] Output: Hand position information, movement history

[1078] Specific behavior:

[1079] The user operates the AR device and touches a virtual object (e.g., the head of a VR character) through the screen. The camera and sensors on the AR device capture this action in real time. Specifically, the position information and movement history of the hands and fingers are acquired and converted into data.

[1080] Step 2:

[1081] Sending data and receiving it on the server

[1082] Input: Hand position information, movement history

[1083] Output: Sending operation information data, storing received data

[1084] Specific behavior:

[1085] The AR device sends the hand position information and movement history to the server. The data is transferred via the network, and the server receives the data and temporarily stores it in a data queue.

[1086] Step 3:

[1087] Data analysis and behavioral inference

[1088] Input: Operation information data

[1089] Output: Estimated result of the action

[1090] Specific behavior:

[1091] The server uses the generated AI model to analyze the received operation information data. Specifically, it analyzes the trajectory, speed, and movement patterns of the hand movement to infer actions such as "stroking." For example, from an analysis of the smooth movement of the hand and its speed, it can infer with a high degree of probability that the action is "stroking."

[1092] Step 4:

[1093] Generate and send feedback

[1094] Input: Estimated result of the action

[1095] Output: Feedback data

[1096] Specific behavior:

[1097] The server generates appropriate feedback based on the action estimation. For example, a vibration pattern or visual feedback (animation) corresponding to the action of "stroking" is generated. The generated feedback data is sent to the VR device.

[1098] Step 5:

[1099] Applying Feedback

[1100] Input: Feedback data

[1101] Output: Tactile and visual feedback to the user

[1102] Specific behavior:

[1103] The VR device responds to the user based on the feedback data received from the server. The HMD generates appropriate vibrations and displays animations of the character being stroked in the VR space. For example, the visual effect of a character's head being stroked along with a slight vibration gives the user the sensation of actually stroking the character.

[1104] (Application example 1)

[1105] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1106] While the direct interaction between customers and store staff forms part of the value of the shopping experience in a physical store, there are limitations in that not all customers can receive support at the same time. Another issue is that when store staff provide support remotely, the information and experience customers receive is limited. This can lead to customers not being able to fully understand how to use a product or its features.

[1107] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1108] In this invention, the server includes: means for transmitting operation information acquired from the AR device to the server; means including a generative AI model that analyzes the operation information and infers actions; means for generating feedback based on the inferred actions and transmitting it to the VR device; means for receiving the feedback and providing tactile or visual feedback to the VR user; means for a customer to wear AR glasses in a physical store and for a remote store staff member to provide product operation instructions using VR gloves; and means for the server to analyze the customer's behavior, generate appropriate feedback, and display it on the AR glasses. This allows customers to intuitively and effectively understand how to use and use products while providing valuable remote support.

[1109] "Interaction between different dimensions" refers to the interaction and information exchange between users that takes place between a virtual space, which is different from the physical space, and a real space using virtual reality or augmented reality.

[1110] An "AR terminal" is a device that uses augmented reality technology to overlay digital information on the real world, and is equipped with a camera and sensors.

[1111] "Operation information" refers to data acquired by the AR device when it recognizes operations such as user movements and gestures.

[1112] A "server" is a high-performance computer system that communicates with various terminals via a network and analyzes data.

[1113] A "generative AI model" is an algorithm or model that uses artificial intelligence to analyze data and infer user operations and behavior.

[1114] A "VR terminal" is a device that uses virtual reality technology to provide users with an immersive virtual space, and includes head-mounted displays and haptic devices.

[1115] "Feedback" refers to the tactile or visual response that is generated in response to a user's operation or action.

[1116] A "brick and mortar store" is a commercial establishment that exists in a physical location and can be visited in person by customers.

[1117] "AR glasses" are glasses-type devices that use augmented reality technology, allowing users to visually view digital information overlaid on the real world.

[1118] "Remote store staff" refers to store staff who are physically located in a different location from the physical store but who support customers virtually.

[1119] "VR gloves" are glove-shaped devices that allow users to operate the device with their hands in a virtual reality world and provide haptic feedback.

[1120] "Analyzing customer behavior" means collecting data on the operations and actions performed by customers, and analyzing that data to infer specific actions.

[1121] This invention provides a system that uses an AR terminal, a server, and a VR terminal to improve the shopping experience in a physical store. This system allows customers to wear AR glasses in the physical store, and remote store staff can use VR gloves to provide instructions on how to operate products.

[1122] First, a customer wears AR glasses and walks around a physical store. The AR device (e.g., a glasses-type device that uses augmented reality technology) captures customer operation information through cameras and sensors. For example, when a customer looks at a product or makes a gesture of reaching out, that operation information is acquired in real time.

[1123] This operation information is then sent to a server and stored in a data queue. The server is a high-performance computer system (e.g., an AWS EC2 instance) that analyzes the received operation information using a generative AI model (e.g., OpenAI's GPT-4). Specifically, the server analyzes hand movements, gaze direction, and movement patterns to infer the action of "selecting a product."

[1124] Based on the analyzed actions, the server generates feedback. A remote store staff member wearing VR gloves (e.g., a glove-type device for operating with hands in a virtual reality world) can use the feedback to demonstrate to the customer how to operate the product and its characteristics. The server generates appropriate feedback (visual and tactile responses) and displays it on the AR glasses, allowing the customer to experience the store staff as if they were right in front of them.

[1125] Here is one concrete example:

[1126] 1. A customer puts on AR glasses in a physical store and looks at a specific product.

[1127] 2. The AR device captures the eye movements and hand gestures and sends the operation information to the server.

[1128] 3. The server's generated AI model analyzes the operation information and infers the action of "selecting a product."

[1129] 4. The server sends notifications to remote store staff based on the analysis results.

[1130] 5. Remote store staff will use VR gloves to virtually demonstrate how to operate the product.

[1131] 6. Feedback is displayed on the customer's AR glasses, allowing them to visually confirm how to use the product and its features.

[1132] Example prompts for generative AI models:

[1133] Analyze the following input data and infer the actions taken by the customer (product selection, inquiry, etc.).

[1134] Input data: { "Action": "Reach out", "Target": "Product code 12345", "Timestamp": "2023-10-01T08:00:00Z"}

[1135] Please output the analysis results in the following format:

[1136] {

[1137] "Action": "Product Selection",

[1138] "Product Code": "12345",

[1139] "Recommended feedback": "View a demonstration of how to use the product"

[1140] }

[1141] The invention makes the in-store shopping experience more intuitive and effective, allowing customers to understand the usage and characteristics of products with real-time support from remote store staff.

[1142] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1143] Step 1:

[1144] The AR device (AR glasses) captures customer operation information. Specifically, it uses cameras and sensors to detect the customer's gaze and hand movements and collects that data. The input is the customer's gaze direction and hand gestures, and the output is data that compiles this operation information.

[1145] Step 2:

[1146] The AR device sends the captured operation information to the server. Specifically, the operation information is transferred to the server in real time via the network. The input is the operation information data obtained in step 1, and the output is the operation information data received by the server.

[1147] Step 3:

[1148] The server stores the received operation information in a data queue. Specifically, the server temporarily saves the operation information and prepares it for analysis. The input is the operation information data sent from the AR terminal, and the output is the operation information stored in the data queue.

[1149] Step 4:

[1150] The server's generative AI model analyzes the operation information. Specifically, the operation information is input into a generative AI model (e.g., GPT-4) to infer the customer's actions. In this process, hand movements and gaze patterns are recognized to infer specific actions (e.g., product selection). The input is the operation information stored in the data queue, and the output is the inferred action result.

[1151] Step 5:

[1152] The server generates feedback based on the inference results. Specifically, it generates feedback data (visual and tactile responses) according to the inferred actions and sends a notification to the remote store staff. The input is the inference results of the generative AI model, and the output is feedback data sent to the remote store staff and the VR gloves.

[1153] Step 6:

[1154] Remote store staff use VR gloves to virtually demonstrate to customers how to operate products. Specifically, they operate the products through the VR gloves and send the operation information to the server. The input is a feedback notification from the server, and the output is feedback data as a demonstration of the operation.

[1155] Step 7:

[1156] The server analyzes the operation information of the remote store staff and generates feedback. Specifically, it analyzes the operation information of the remote store staff and generates visual and tactile feedback to be displayed on the AR terminal (AR glasses). The input is the operation information from the remote store staff, and the output is feedback data sent to the AR glasses.

[1157] Step 8:

[1158] The AR device (AR glasses) provides the customer with the feedback received from the server. Specifically, it displays guides and arrows that visually show how to use the product, and provides tactile feedback as needed. The input is the feedback data from the server, and the output is the visual and tactile response that the customer actually experiences.

[1159] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1160] This invention provides a system for realizing interactions across different dimensions. This system consists of an AR terminal, a server, a VR terminal, and an emotion engine, and its main processing steps include capturing operation information, transmitting data, analyzing data, recognizing emotions, generating feedback, and applying feedback.

[1161] System configuration

[1162] AR device

[1163] AR devices are devices that use augmented reality technology to display digital information overlaid on the real world. These devices are equipped with cameras and sensors that capture the position and movement history of the AR user's hands and fingers in real time.

[1164] server

[1165] The server is a high-performance computer system that communicates with the AR and VR devices via a network. The server is equipped with a generative AI model and an emotion engine, and analyzes the operation information received from the AR device to estimate the user's actions and emotions.

[1166] VR device

[1167] A VR device is a device that uses virtual reality technology to provide an immersive virtual space. Its main components include a head-mounted display (HMD) and a vibration device. The VR device provides tactile or visual responses to the VR user based on feedback signals sent from the server.

[1168] Emotion Engine

[1169] The emotion engine is a software component that recognizes the emotions of AR users from their voices, facial expressions, gestures, etc. This allows it to generate more accurate feedback by combining it with operational information.

[1170] Program processing and examples

[1171] AR device operation capture

[1172] The AR device captures gestures made by the AR user toward the VR user's character. For example, if the AR user uses a tablet to make a gesture of patting the VR user's character's head through the screen, the sensor will capture the hand movement and position information.

[1173] Sending and Receiving Data

[1174] The AR device sends the captured operation information to the server, using a real-time communication protocol (e.g., WebSocket) to transmit data with low latency.

[1175] Data analysis and behavioral inference

[1176] The server uses the generated AI model to analyze the received operation information. Specifically, it infers the AR user's actions from hand movements and position information. In this process, it uses pre-trained gesture classes to classify actions such as "stroking" and "hitting."

[1177] emotion recognition

[1178] The server's emotion engine analyzes the AR user's voice, facial expressions, and gestures to recognize their emotions. For example, if the AR user is smiling and patting someone on the head, the emotion of "joy" is recognized.

[1179] Generate and send feedback

[1180] The server generates a feedback signal based on the analysis results and the recognized emotion. For example, if the action of "stroking" and the emotion of "joy" are inferred, the server generates an appropriate vibration pattern and visual feedback that conveys joy, and sends that signal to the VR device.

[1181] Applying Feedback

[1182] The VR device receives the feedback signal sent from the server. The received signal is used as an instruction to provide tactile and visual responses to the VR user. Specifically, the VR user's head-mounted device (HMD) generates vibrations, and animations of being petted or graphics conveying pleasure are displayed in the VR space.

[1183] Specific examples

[1184] For example, suppose an AR user uses a tablet to make a gesture of patting a VR user's head while smiling. The AR device captures this gesture and facial expression and sends it to the server. The server's generative AI model analyzes this data to infer the action of "patting," and the emotion engine recognizes the emotion of "joy." The server then generates vibration and visual feedback to convey joy and sends it to the VR device. The VR device receives this feedback and generates a slight vibration in the VR user's HMD, while simultaneously displaying a graphic that expresses joy along with an animation of the pat. This allows the VR user to experience the sensation of actually being petted while also sensing the AR user's emotions.

[1185] This system allows AR and VR users to interact intuitively and emotionally, even though they are in different dimensions.

[1186] The processing flow will be explained below.

[1187] Step 1:

[1188] The AR device captures the gestures made by the AR user. Specifically, the tablet's camera and sensors capture the AR user's hand position (x, y, z coordinates) and movement (velocity, acceleration) in real time. This information is captured frame by frame.

[1189] Step 2:

[1190] The AR device sends the acquired gesture data to the server using a real-time communication protocol (e.g., WebSocket) to transmit the data with low latency.

[1191] Step 3:

[1192] The server receives the gesture data sent from the AR device and temporarily stores the received data in a queue.

[1193] Step 4:

[1194] The server uses an emotion engine to recognize the emotions of the AR user. For example, the server analyzes the voice and facial expression data received from the AR device to identify emotions such as "joy" or "sadness."

[1195] Step 5:

[1196] The server analyzes the received gesture data using a generative AI model to infer the AR user's actions. For example, it can infer the action of "stroking" from hand movements and position information.

[1197] Step 6:

[1198] The server generates a feedback signal based on the analysis results and the recognized emotion. For example, if the action of "stroking" and the emotion of "joy" are inferred, the server generates a specific vibration pattern and visual feedback that conveys joy.

[1199] Step 7:

[1200] The server transmits the generated feedback signal to the VR device using a real-time communication protocol with low latency.

[1201] Step 8:

[1202] The VR device receives the feedback signals sent from the server, which are used to provide haptic and visual responses.

[1203] Step 9:

[1204] The VR device applies a reaction to the VR user based on the received feedback signal. Specifically, it generates vibrations in the VR user's HMD and displays animations of being petted and graphics that convey pleasure in the VR space.

[1205] Step 10:

[1206] The VR user feels the AR user stroking them through vibrations in the head and visual feedback, and can also sense the AR user's emotions based on the feedback.

[1207] As a specific example, consider a scene in which an AR user uses a tablet to make a gesture of patting a VR user's head, smiling while doing so. The AR device captures the gesture and facial expression and sends the data to the server. The server's emotion engine recognizes the smile as the emotion of "happiness," and the generative AI model infers the action of "patting." The server then generates a feedback signal based on the emotion and action and sends it to the VR device. The VR device receives this feedback, generates a slight vibration in the VR user's HMD, and displays an animation expressing happiness. This gives the VR user the feeling of actually being petted, and also allows them to sense the AR user's emotion of happiness.

[1208] Through these steps, the present invention realizes intuitive and emotionally rich interaction that transcends the dimensions of AR and VR.

[1209] Example 2

[1210] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1211] In conventional AR and VR systems, it has been difficult to achieve intuitive and emotion-based interaction across different dimensions. In particular, there has been a lack of systems that can analyze users' gestures and actions in real time and provide appropriate feedback based on that analysis. In addition, there are limited means to accurately recognize users' emotions and generate feedback based on those emotions.

[1212] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1213] In this invention, the server includes a means for transmitting operation information acquired from the AR terminal to the server, a means including a generative AI model for analyzing the operation information and inferring an action, an emotion recognition means for analyzing the operation information and the user's facial expressions and voice and recognizing emotions, a means for generating feedback based on the inferred action and the recognized emotion and transmitting it to the VR terminal, and a means for receiving the feedback and providing a tactile or visual response to the VR user, thereby enabling intuitive and emotional interactions across different dimensions.

[1214] An "AR terminal" is a device that uses augmented reality technology to overlay digital information on the real world.

[1215] "Operation information" is data on the movements and positions of the user's hands and fingers acquired from the AR terminal.

[1216] A "server" is a high-performance computer system that communicates with AR and VR terminals to analyze and process data.

[1217] A "generative AI model" is a model that uses machine learning algorithms to analyze user gesture data and classify and infer specific actions.

[1218] An "emotion recognition means" is a system or algorithm that analyzes the user's voice, facial expressions, gestures, etc. to recognize their emotions.

[1219] "Feedback" refers to the tactile or visual response generated based on actions and perceived emotions.

[1220] A "VR terminal" is a device that uses virtual reality technology to provide an immersive virtual space, and includes a head-mounted display (HMD) and a vibration device.

[1221] "Haptic feedback" refers to a physical stimulus (e.g., vibration) that provides a tactile sensation to the user.

[1222] "Visual response" refers to providing feedback to the user through visual information, including animations and graphics.

[1223] "Interaction across different dimensions" refers to users in the real world and the virtual world sharing and interacting with each other's actions and emotions.

[1224] The system of this invention is composed of an AR terminal, a server, a VR terminal, and emotion recognition means. We will explain a specific embodiment that realizes intuitive and emotional interaction between AR users and VR users across different dimensions through the operation of the entire system.

[1225] Hardware and software used

[1226] AR device: An augmented reality device such as Microsoft HoloLens, equipped with a camera and sensors that captures user movements in real time.

[1227] Server: A high-performance computer system (e.g., AWS EC2 instance) that runs the generative AI model and emotion recognition method.

[1228] VR device: A virtual reality device such as the Oculus Rift, including a head-mounted display (HMD) and a vibration device.

[1229] Emotion recognition method: Uses Microsoft Azure's Emotion API, etc.

[1230] Data processing and calculation

[1231] When a user uses an AR device to make a gesture (e.g., stroking the head) to the VR user's character, the camera and sensors of the AR device capture this gesture and obtain the position information and movement history of the hand and fingers in real time.

[1232] The operation information acquired by the AR device is sent to the server using a real-time communication protocol (e.g., WebSocket). The server receives this data and begins analyzing it using the generative AI model.

[1233] The server uses a generative AI model (for example, a custom model based on TensorFlow) to analyze hand movements and position information and infer actions such as "stroking." Along with the analyzed action data, the server captures the user's facial expressions and voice, and recognizes their emotions using emotion recognition means.

[1234] For example, if the user is smiling and stroking the user's head, the emotion recognition means recognizes the emotion as "joy." The server generates a feedback signal based on the analysis of the user's behavior and emotion.

[1235] The feedback signals include haptic feedback (e.g., vibration of the HMD) and visual feedback (e.g., animation of being petted and graphics expressing pleasure). The server generates the feedback signals and sends them to the VR device.

[1236] The VR device receives the feedback signal sent from the server and provides specific tactile and visual responses to the VR user, such as generating a slight vibration in the HMD and displaying a graphic indicating pleasure along with an animation of the user being stroked.

[1237] Specific examples

[1238] For example, suppose a user uses a tablet to make a gesture of patting a VR user's head while smiling. The AR device captures this gesture and facial expression and sends it to the server. The server's generative AI model analyzes this data to infer the action of "patting," and the emotion recognition means recognizes the emotion of "joy." The server then generates vibration and visual feedback to convey joy and sends it to the VR device. The VR device receives this feedback and generates a slight vibration in the HMD, while simultaneously displaying a graphic that expresses joy along with an animation of the user being stroked. This allows the VR user to experience the sensation of actually being stroked while also sensing the user's emotions.

[1239] Prompt Sentence Examples

[1240] "Imagine an AR user using a tablet to smile and stroke the head of the VR user character through the screen. The AR device captures this gesture and facial expression and sends it to the server in real time. The server's AI model analyzes it and detects the 'stroking' gesture, and the emotion recognition means recognizes the emotion 'happiness'. As a result, the server generates appropriate haptic and visual feedback and sends it to the VR device. The VR device receives this feedback and vibrates the HMD, displaying an animation of the user being stroked and a graphic of happiness."

[1241] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1242] Step 1:

[1243] The user makes a gesture using the AR device. Specifically, the user uses a tablet to pat the head of the VR user's character. At this time, the camera and sensors of the AR device capture the position information and movement history of the hand and fingers in real time. The input is the user's gesture, and the output is real-time position information and movement history.

[1244] Step 2:

[1245] The operation information captured by the device (AR device) is sent to the server via a real-time communication protocol (e.g., WebSocket). Specifically, the AR device forms the position information and movement history of the hands and fingers it has acquired into a data packet and sends it to the server. The input is the real-time position information and movement history, and the output is the sent data packet.

[1246] Step 3:

[1247] The server receives data packets sent from the AR device. Specifically, the server receives the data packets via WebSocket, decodes the packets, and extracts the operation information. The input is the sent data packet, and the output is the operation information.

[1248] Step 4:

[1249] The operation information received by the server is analyzed using a generative AI model. Specifically, the generative AI model (e.g., a custom model based on TensorFlow) analyzes the position information and movement history of the hand and fingers, and infers that the user's gesture is a specific action, such as "stroking." This analysis is performed based on pre-trained gesture classes. The input is operation information, and the output is data on the inferred action.

[1250] Step 5:

[1251] The server uses an emotion recognition means to recognize the user's emotions. Specifically, it analyzes the user's voice, facial expression, and gesture information, and the emotion recognition means (e.g., Microsoft Azure's Emotion API) detects emotions such as "joy." The input is voice, facial expression, and gesture information, and the output is recognized emotion data.

[1252] Step 6:

[1253] The server generates a feedback signal based on the estimated action and the recognized emotion. Specifically, the server combines the action of "stroking" with the emotion of "joy" to generate a signal including haptic feedback (e.g., HMD vibration) and visual feedback (e.g., an animation of being stroked and a graphic expressing joy). The input is action data and emotion data, and the output is a feedback signal.

[1254] Step 7:

[1255] The server generates feedback signals and sends them to the VR device. Specifically, the feedback signals are formed as data packets and sent to the VR device via WebSocket. This provides sensory information to the VR user. The input is the generated feedback signal, and the output is the transmitted data packet.

[1256] Step 8:

[1257] The terminal (VR terminal) receives the feedback signal sent from the server and provides specific tactile and visual responses. Specifically, it generates a slight vibration in the HMD and displays an animation of being stroked and a graphic that expresses pleasure in the VR space. The received feedback signal is the input, and the tactile and visual feedback is provided as the output.

[1258] (Application example 2)

[1259] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1260] It solves the challenge of realizing real-time, emotional interaction between different dimensions, namely augmented reality (AR) and virtual reality (VR), and also improves the customer experience in virtual stores, enabling personalized shopping experiences.

[1261] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means including a generative artificial intelligence model that analyzes operation information acquired from the AR terminal and infers an action, means for generating feedback based on the inferred action and the recognized emotion and transmitting the feedback to the VR terminal, and means for the feedback to include visual and tactile feedback that personalizes the shopping experience in the virtual space. This enables intuitive and emotionally rich interactions between different dimensions and makes it possible to provide a personalized customer experience in the virtual store.

[1262] "Different dimensions" refers to the different realms of existence of augmented reality (AR), which overlays digital information on the real world, and virtual reality (VR), which provides a virtual space.

[1263] A "generative artificial intelligence model" is a modeling method based on machine learning and deep learning that is used to analyze received operational information and infer actions and emotions.

[1264] "Operation information" refers to data such as hand and finger position information and movement history obtained from the AR device.

[1265] "Feedback" refers to a reaction signal generated based on the inferred action and the recognized emotion, to provide the user with a tactile or visual response.

[1266] "Virtual space" refers to a virtual environment or world that can be experienced through a VR device.

[1267] "Gesture data" refers to data related to hand movements and gestures made by a user.

[1268] "Facial expression data" refers to data relating to the user's facial expressions.

[1269] "Promotion information" is information related to sales and special campaigns for specific products.

[1270] "Visual feedback" refers to information provided to a user through their sense of sight, including graphics and animations displayed on a display.

[1271] An "AR terminal" refers to a device that uses augmented reality technology to overlay digital information on the real world.

[1272] A "VR terminal" refers to a device that uses virtual reality technology to provide an immersive virtual space.

[1273] System configuration

[1274] A system for realizing this application example uses the following hardware and software.

[1275] Hardware

[1276] 1. AR device: A device (e.g., smart glasses) that uses augmented reality technology to overlay digital information on the real world.

[1277] 2. VR terminal: A device that uses virtual reality technology to provide an immersive virtual space (e.g., a head-mounted display).

[1278] 3. Server: A high performance computer server.

[1279] software

[1280] 1. Generative AI model: An artificial intelligence model (e.g., TensorFlow or PyTorch) for analyzing hand movements and facial expressions.

[1281] 2. Emotion engine: Software that analyzes voice and facial expression data to recognize emotions (e.g., Google Cloud Natural Language API, Amazon Comprehend).

[1282] 3. Communication protocol: A protocol for communicating data in real time (e.g., WebSocket).

[1283] What the program does

[1284] This system processes data and realizes interactions in the following steps:

[1285] AR device processing

[1286] AR devices are equipped with sensors to capture the user's hand and finger movements and facial expressions. In particular, the AR device acquires the captured gesture and facial expression data in real time and transmits it to a server. For example, if a user points at a product using smart glasses and their facial expression shows delight, that information will be captured.

[1287] Server Processing

[1288] The server is equipped with a generative AI model and emotion engine. The server analyzes the operation information sent from the AR device and estimates the action and emotion. Specifically, it analyzes hand movements and position information to estimate the "pointing" action and further recognizes the emotion of "happiness" from the user's facial expression.

[1289] Generate and send feedback

[1290] Based on the analysis results, the server generates appropriate feedback, including visual and tactile responses. For example, if the estimated action is "pointing" and the emotion is "joy," the server generates visual feedback containing detailed product information and promotional information and sends it to the VR device.

[1291] Applying Feedback

[1292] The VR device responds appropriately in the virtual space based on the received feedback signal, providing visual feedback to the user by displaying product guides or spontaneously displaying promotional information about specific products.

[1293] Examples and prompts

[1294] For example, if a customer uses smart glasses to look at a product in a virtual store, makes a gesture of touching it with their hands, and shows an excited expression, this information is captured and sent to the server. The server analyzes this data and recognizes the behavior of "showing interest in a particular product" and the emotion of "excitement." It then generates feedback on the VR device that displays detailed product information and special discount promotions. The example prompt sentences used by the generative AI model are as follows:

[1295] User is viewing and interacting with a product. Capture the movements and facial expressions, and provide appropriate promotional feedback.

[1296] This makes the user's shopping experience in a virtual store more intuitive and emotionally satisfying.

[1297] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1298] Step 1:

[1299] The AR device captures the user's hand and finger movements and facial expressions in real time. Gesture and facial expression data obtained from sensors is received as input and sent to a server. Specifically, the smart glasses' cameras and sensors capture and measure the user's movements, and the data is processed and compiled as operation information. Operation information includes the position and movement of the hands, facial expressions, etc. This information is sent to the server in real time via WebSocket.

[1300] Step 2:

[1301] The server receives operation information sent from the AR device. The received data includes the user's gestures and facial expression data. The server analyzes this data using a generative AI model and an emotion engine. Specifically, the generative AI model analyzes hand movements and position information to infer actions, and the emotion engine analyzes facial expression data and voice to recognize emotions. As a result of this analysis, the server outputs an action estimation result (e.g., "pointing") and an emotion recognition result (e.g., "excitement").

[1302] Step 3:

[1303] The server generates appropriate feedback based on the estimated action and the recognized emotion. It receives the action estimation results and emotion recognition results as input and generates feedback data accordingly. For example, if the action of "pointing" and the emotion of "excitement" are recognized, the server visually displays detailed information about a product and creates feedback including promotional information related to that product. The generated feedback data is sent to the VR device.

[1304] Step 4:

[1305] The VR device receives the feedback data sent from the server. The received data includes visual feedback and haptic feedback. Specifically, the VR device displays detailed product information and promotional information in the virtual space as visual feedback, and generates device vibrations and other haptic feedback. This process allows the user to experience real-time feedback in the virtual space.

[1306] Step 5:

[1307] Users experience feedback provided through the VR device. Specifically, when a user points at a product in the virtual space, detailed information about that product is displayed. Also, if a user shows interest in a particular product, related promotional information is displayed. This makes the shopping experience in the virtual store more intuitive and emotionally satisfying.

[1308] The above steps enable interaction across different dimensions and improve customer experience in virtual stores.

[1309] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1310] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1311] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1312] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1313] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1314] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1315] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1316] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1317] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1318] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1319] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1320] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1321] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1322] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1323] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1324] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1325] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1326] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1327] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1328] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1329] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1330] The following is further disclosed regarding the above embodiment.

[1331] (Claim 1)

[1332] To realize interactions between different dimensions,

[1333] means for transmitting operation information acquired from the AR terminal to a server;

[1334] means including a generative AI model that analyzes the operation information and infers an action;

[1335] means for generating feedback based on the estimated action and transmitting the feedback to the VR terminal;

[1336] means for receiving said feedback and providing a haptic or visual response to the VR user;

[1337] A system including:

[1338] (Claim 2)

[1339] 10. The system of claim 1, wherein the generative AI model includes means for analyzing AR user gesture data and classifying specific actions.

[1340] (Claim 3)

[1341] The system of claim 1 , wherein the feedback comprises means for providing vibration feedback.

[1342] (Claim 4)

[1343] The system of claim 1 , wherein the AR terminal is a tablet or a smart device.

[1344] (Claim 5)

[1345] The system of claim 1 , wherein the VR terminal includes a head-mounted display.

[1346] (Claim 6)

[1347] The system of claim 1 , wherein the operation information includes hand position information and movement history.

[1348] (Claim 7)

[1349] The system of claim 1 , wherein the feedback includes providing an animation as a visual response.

[1350] "Example 1"

[1351] (Claim 1)

[1352] To realize interactions between different dimensions,

[1353] means for transmitting operation information acquired from the AR terminal to a server;

[1354] means including a generative AI model that analyzes the operation information and infers an action;

[1355] means for generating feedback based on the estimated action and transmitting the feedback to the VR terminal;

[1356] means for receiving said feedback and providing a haptic or visual response to the VR user;

[1357] A means for capturing gestures in real time using the camera and sensors of the AR device;

[1358] A means of reproducing feedback using a head-mounted display or vibration device,

[1359] A system including:

[1360] (Claim 2)

[1361] 10. The system of claim 1, wherein the generative AI model includes means for analyzing the AR user's hand position information and movement history and classifying specific actions.

[1362] (Claim 3)

[1363] The system of claim 1 , wherein the feedback comprises means for providing vibration feedback and visual animation feedback.

[1364] "Application Example 1"

[1365] (Claim 1)

[1366] To realize interactions between different dimensions,

[1367] means for transmitting operation information acquired from the AR terminal to a server;

[1368] means including a generative AI model that analyzes the operation information and infers an action;

[1369] means for generating feedback based on the estimated action and transmitting the feedback to the VR terminal;

[1370] means for receiving said feedback and providing a haptic or visual response to the VR user;

[1371] A method in which customers use AR glasses in a physical store and remote store staff use VR gloves to provide product operation instructions;

[1372] A server analyzes the customer's behavior, generates appropriate feedback, and displays it on the AR glasses.

[1373] A system including:

[1374] (Claim 2)

[1375] 10. The system of claim 1, wherein the generative AI model includes means for analyzing AR user gesture data and classifying specific actions.

[1376] (Claim 3)

[1377] 10. The system of claim 1, wherein the feedback comprises means for providing vibration feedback.

[1378] "Example 2: Combining Emotion Engines"

[1379] Claims

[1380] (Claim 1)

[1381] means for transmitting operation information acquired from the AR terminal to a server;

[1382] means including a generative AI model that analyzes the operation information and infers an action;

[1383] emotion recognition means for analyzing the operation information and the user's facial expression and voice to recognize emotions;

[1384] means for generating feedback based on the estimated action and the recognized emotion and transmitting the feedback to the VR terminal;

[1385] means for receiving said feedback and providing a haptic or visual response to the VR user;

[1386] A system including:

[1387] (Claim 2)

[1388] 10. The system of claim 1, wherein the generative AI model includes means for analyzing AR user gesture data and classifying specific actions.

[1389] (Claim 3)

[1390] The system of claim 1 , wherein the feedback comprises means for providing vibration feedback.

[1391] (Claim 4)

[1392] The system of claim 1 , wherein the emotion recognition means includes means for recognizing emotions from a user's voice, facial expressions, and gestures.

[1393] (Claim 5)

[1394] The system of claim 1 , wherein the feedback includes a visual means for conveying a user's emotion.

[1395] "Application example 2 when combining emotion engines"

[1396] (Claim 1)

[1397] To realize interactions between different dimensions,

[1398] means for transmitting operation information acquired from the AR terminal to a server;

[1399] means including a generative artificial intelligence model that analyzes the operation information and infers an action;

[1400] means for generating feedback based on the estimated action and the recognized emotion and transmitting the feedback to the VR terminal;

[1401] means for receiving said feedback and providing a haptic or visual response to the VR user;

[1402] said feedback including visual and tactile feedback to personalize the shopping experience within the virtual space;

[1403] A system including:

[1404] (Claim 2)

[1405] 10. The system of claim 1, wherein the generative artificial intelligence model includes means for analyzing gesture and facial expression data of an AR user and classifying specific actions and emotions.

[1406] (Claim 3)

[1407] The system of claim 1 , wherein the feedback comprises means for providing visual feedback including promotional information and product details. [Explanation of symbols]

[1408] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. To realize interactions between different dimensions, means for transmitting operation information acquired from the AR terminal to a server; A means including a generative AI model that analyzes the operation information and infers an action; means for generating feedback based on the estimated action and transmitting the feedback to the VR terminal; means for receiving said feedback and providing a haptic or visual response to the VR user; A system including:

2. The system of claim 1 , wherein the generative AI model includes means for analyzing gesture data of an AR user and classifying specific actions.

3. The system of claim 1 , wherein the feedback comprises means for providing vibration feedback.

4. The system of claim 1 , wherein the AR terminal is a tablet or a smart device.

5. The system of claim 1 , wherein the VR terminal includes a head-mounted display.

6. The system of claim 1 , wherein the operation information includes hand position information and movement history.

7. The system of claim 1 , wherein the feedback includes means for providing an animation as a visual response.

Images