System

Abstract

A system is provided.SOLUTION: A system includes a customization means for customizing an avatar and a virtual pet of a user, a data storage means for storing customization data generated by the customization means, a data sharing means for sharing the customization data stored in the storage means with other users, a viewpoint obtaining means for obtaining viewpoint information of the user, a drawing means for drawing the virtual pet and the avatar using a generated AI based on the viewpoint information obtained by the viewpoint obtaining means, and a display means for displaying the avatar and the virtual pet drawn by the drawing means in a field of view of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional augmented reality (AR) technology allows individual users to enjoy AR content only within their own field of vision, making it difficult to share the experience with other users. Furthermore, the placement of AR content in real-world locations and the methods for enjoying collaborative experiences are limited. The present invention aims to provide a system that enables multiple users wearing the same AR device to share the same AR content in real time and enjoy the experience together. Furthermore, the present invention aims to realize a more integrated AR experience by allowing users to share their customized avatars and virtual pets with other users, providing an experience that feels as if they are actually there. [Means for solving the problem]

[0005] The present invention provides the following means: A customization means for allowing a user to customize their own avatar and virtual pet, and a data storage means for storing customization data generated by the customization means. The system further includes a data sharing means for sharing the customization data stored in the storage means with other users, and a viewpoint acquisition means for acquiring user viewpoint information. The system also includes a drawing means for drawing a virtual pet and avatar using a generation AI based on the viewpoint information acquired by the viewpoint acquisition means, and a display means for displaying the avatar and virtual pet drawn by the drawing means in the user's field of view. This system allows users to share the same AR content with other users in real time and engage in a collaborative experience.

[0006] The data sharing means can distribute customized data in real time to other users wearing AR devices in the same space. Furthermore, the rendering means includes a means for sharing information about AR content installed in a specific location and allowing multiple users to share the experience in real time. This enhances interaction between users and provides a more immersive AR experience.

[0007] The term "user" refers to an individual or a corporation that wears and uses the AR device of the present invention.

[0008] "Customization means" refers to the interface and processing means by which a user can set and edit the appearance and behavior of their avatar or virtual pet.

[0009] "Customization data" refers to data including setting information for an avatar or virtual pet that a user creates using customization means.

[0010] The "data storage means" refers to a storage for saving the customization data generated by the customization means and a means for managing the storage.

[0011] The "data sharing means" refers to a communication means and a processing means for sharing the saved customized data with other users wearing the same AR device.

[0012] "Viewpoint acquisition means" refers to a camera, a sensor, and a processing means for acquiring the user's current viewpoint information (position, direction, etc.).

[0013] "Rendering means" refers to processing means for rendering a virtual pet and an avatar in real time using a generating AI based on viewpoint information acquired by viewpoint acquisition means.

[0014] "Generative AI" refers to algorithms and their implementation methods for generating realistic depictions of virtual pets or avatars using artificial intelligence techniques.

[0015] "Display means" refers to a display and its control means for displaying the avatar and virtual pet generated by the drawing means in the user's field of view.

[0016] "AR content" refers to virtual content (e.g., a virtual crane game) that is displayed in real space using augmented reality technology.

[0017] A "collaborative experience" refers to an experience in which multiple users simultaneously share the same AR content and interact with each other in real time. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The purpose of the experience sharing AR system of the present invention is to enable multiple users to share the same AR content in real time and interact with each other. An embodiment of this system will be described in detail below.

[0040] overview

[0041] The AR device (terminal) worn by the user has the means to customize the user's avatar and virtual pet and share it with other users, and uses generation AI to render the virtual pet and avatar in real time based on the user's viewpoint, providing the user with an immersive experience.

[0042] Program processing

[0043] 1. Customize your avatar and virtual pet

[0044] The terminal provides the user with an interface for customizing their avatar and virtual pet.

[0045] The user selects the clothes, hats, accessories, type and characteristics of the virtual pet.

[0046] Customized data is generated within the device.

[0047] The terminal transmits the generated customization data to the server.

[0048] 2. Data storage and sharing

[0049] The server stores the received customization data in a database.

[0050] The server distributes the customization data to other users wearing the same AR device.

[0051] The other user's device renders the user's avatar and virtual pet based on the received customization data.

[0052] 3. Real-time drawing

[0053] The device acquires the user's viewpoint information using a camera or sensor.

[0054] The terminal transmits the acquired viewpoint information to the server.

[0055] The server uses a generative AI based on the received viewpoint information to create instructions for drawing a virtual pet or avatar in real time.

[0056] The server sends drawing instructions to the terminal.

[0057] The terminal displays the avatar or virtual pet in the user's field of view based on the received drawing instructions.

[0058] 4. Providing a collaborative experience

[0059] The server distributes information about AR content installed in a specific location (e.g., advertising space on a train).

[0060] The device draws virtual content (e.g., a crane game) at that location based on the received AR content information.

[0061] Users can interact with AR content and share the experience with other users in real time.

[0062] Specific examples

[0063] Example 1: User A and User B are on the same train

[0064] First, user A customizes his or her avatar with a blue shirt and jeans and a virtual pet with a black cat. Once the customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to user B's device. User B's device then renders user A's avatar and the black cat in real time. Similarly, if user B changes his or her avatar to a red hat and sets his or her virtual pet to a white dog, the data is shared using the same procedure.

[0065] Example 2: Virtual "crane game" on a train

[0066] The server sets up a virtual "crane game" in the advertising space on the train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B can participate in the same game and enjoy a collaborative experience by operating the crane in real time.

[0067] In this way, the system of the present invention allows users to share AR content with other users in real time and enjoy an immersive collaborative experience.

[0068] The processing flow will be explained below.

[0069] Below, the program processing of the experience-sharing AR system is explained in concrete steps.

[0070] Customize your avatar and virtual pet

[0071] Step 1:

[0072] The user launches the "Avatar Connect" application.

[0073] The user opens the application and is presented with a login screen.

[0074] Step 2:

[0075] The user enters their login information and logs in.

[0076] The user enters their account information and clicks the "Login" button.

[0077] The application sends the login information to the server.

[0078] Step 3:

[0079] The server authenticates the user's login information.

[0080] The server checks the database to verify the login information.

[0081] The authentication result is returned to the terminal.

[0082] Step 4:

[0083] The terminal displays a customization screen to the user.

[0084] If authentication is successful, the customization interface will be displayed.

[0085] Users can choose the appearance of their avatar and virtual pet.

[0086] Step 5:

[0087] Users customize their avatars and virtual pets.

[0088] Choose from multiple options (clothes, hats, pet types, etc.).

[0089] Confirm your customizations with the "Save" button.

[0090] Step 6:

[0091] The terminal transmits the customization data to the server.

[0092] The terminal generates customization data and transmits it to the server.

[0093] Data storage and sharing

[0094] Step 7:

[0095] The server stores the customization data in a database.

[0096] The received customization data is recorded in a database.

[0097] Step 8:

[0098] The server distributes the data to other users.

[0099] Data is distributed in real time to other users wearing the same AR device.

[0100] Step 9:

[0101] The terminal receives customization information of other users.

[0102] Based on the customization information received by the terminal, the avatars and virtual pets of other users are drawn.

[0103] Real-time drawing

[0104] Step 10:

[0105] The terminal acquires the user's viewpoint information.

[0106] Cameras and sensors are used to obtain the user's viewpoint information (position, direction).

[0107] Step 11:

[0108] The terminal transmits viewpoint information to the server.

[0109] The acquired viewpoint information is sent to the server in real time.

[0110] Step 12:

[0111] The server creates drawing instructions using a generation AI.

[0112] Based on the received viewpoint information, the generation AI generates drawing instructions for the virtual pet or avatar.

[0113] Sends drawing instructions to the device.

[0114] Step 13:

[0115] The terminal performs real-time drawing.

[0116] Based on instructions received from the server, an avatar or virtual pet is displayed in the user's field of view.

[0117] Providing a shared experience

[0118] Step 14:

[0119] The server distributes information about AR content placed in a specific location.

[0120] Virtual content (such as a crane game) is placed in advertising space in a specific location (e.g., inside a train), and that information is sent to all devices.

[0121] Step 15:

[0122] The terminal receives the content information and renders it.

[0123] Based on the received AR content information, virtual content is drawn at that location.

[0124] Step 16:

[0125] Users can share experiences with AR content.

[0126] Multiple users can simultaneously interact with virtual content and enjoy a collaborative experience in real time.

[0127] Example 1

[0128] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0129] In current augmented reality (AR) technology, it has been difficult for users to share their avatars or virtual characters with other users in real time, and even to share experiences collaboratively. Furthermore, there are insufficient means for users to instantly reflect their customized avatars or virtual characters on other users' screens. As a result, the immersive and interactive nature of the AR experience is diminished, resulting in a decrease in user satisfaction.

[0130] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0131] In this invention, the server includes a customization means for allowing a user to customize his or her own avatar and virtual character, a data storage means for storing customization data generated by the customization means, a data sharing means for sharing the customization data stored in the storage means with other users, a viewpoint acquisition means for acquiring user viewpoint information, a drawing means for drawing a virtual character and an avatar using a generating artificial intelligence based on the viewpoint information acquired by the viewpoint acquisition means, a display means for displaying the avatar and virtual character drawn by the drawing means in the user's field of view, and a collaborative experience means for allowing multiple users to share the same augmented reality content in real time and have a collaborative experience. This allows users to experience their own and other users' customized avatars and virtual characters in real time and enjoy a collaborative interactive AR experience.

[0132] 1. An "avatar" is a digital character that represents a user in a virtual environment.

[0133] 2. "Virtual Character" means a digital pet or character that a user can customize and control or display within a virtual environment.

[0134] 3. "Customization Method" means the interface and functionality that allows a User to customize and change the type, characteristics, clothing, accessories, etc. of their Avatar or Virtual Character.

[0135] 4. "Data Storage Means" means a storage device such as a database or memory that stores information about a user's customized avatar and virtual character.

[0136] 5. "Data Sharing Method" means the communication method and functionality for sharing saved customized data with other users.

[0137] 6. "Point of view acquisition means" refers to devices and technologies for acquiring user's point of view information using cameras and sensors.

[0138] 7. “Generative AI” refers to artificial intelligence techniques and models for rendering avatars or virtual characters in real time based on a user’s perspective.

[0139] 8. "Rendering means" means a graphics engine or software that displays an avatar or virtual character in the user's field of view based on the acquired viewpoint information and customization data.

[0140] 9. "Display means" means the technology and devices for displaying a rendered avatar or virtual character on the display of an AR device worn by a user.

[0141] 10. "Collaborative experience means" refers to functions and technologies that enable multiple users to share the same augmented reality content in real time and interact with it collaboratively.

[0142] The present invention provides a system for customizing and sharing avatars and virtual characters in real time with other users using augmented reality (AR) devices worn by users, thereby enabling users to share interactive and immersive experiences with other users. Detailed embodiments of the present invention are described below.

[0143] System Configuration

[0144] This system mainly consists of a customization means, a data storage means, a data sharing means, a viewpoint acquisition means, a drawing means, and a display means. The specific processing of each means is as follows.

[0145] Hardware and software used

[0146] Device: AR device worn by the user (e.g., AR glasses, smartphone)

[0147] Server: Cloud server or on-premise server

[0148] Generative AI model: GAN (generative artificial network)

[0149] Database: MySQL, PostgreSQL

[0150] Communication technology: HTTP, WebSocket

[0151] Graphics API: OpenGL, DirectX

[0152] Rendering engine: Unity, Unreal Engine

[0153] Program processing

[0154] Avatar / Virtual Character Customization

[0155] The device provides the user with an interface for customizing their avatar and virtual character. The user uses this interface to select the avatar's clothing and hat, as well as the type and characteristics of the virtual character. The selected customization data is generated within the device and saved in JSON format or similar.

[0156] Data storage and sharing

[0157] The generated customization data is sent from the device to a server. The server stores the received data in a database and shares it with other users in real time using a data sharing means. The other users' devices analyze the received data and draw the avatar and virtual character.

[0158] Obtaining viewpoint information

[0159] The device acquires the user's viewpoint information using the built-in camera and sensors. This information includes the user's position, orientation, and gaze point. The acquired viewpoint information is sent to the server.

[0160] Real-time drawing

[0161] The server uses a generative AI model based on the received viewpoint information to generate instructions for rendering the virtual character and avatar in real time, which are then sent to the device, which then displays the avatar and virtual character in the user's field of view.

[0162] Providing a shared experience

[0163] The server also distributes information about augmented reality content installed in specific locations, allowing multiple users to share the same content, such as a virtual "crane game" installed on a train. Users can manipulate this content and enjoy a collaborative experience with other users in real time.

[0164] Specific examples

[0165] Example 1: User A and User B are in the same location

[0166] User A customizes his avatar with a blue shirt and jeans and a virtual character with a black cat. Once customization is complete, his device sends this information to the server. The server shares the data with other users and distributes it to User B's device. User B's device then renders User A's avatar and the black cat in real time. Similarly, if User B changes his avatar to a red hat and sets his virtual character to a white dog, the data is shared using the same procedure.

[0167] Example 2: Virtual "crane game" on a train

[0168] The server sets up a virtual "crane game" in the advertising space on the train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B can participate in the same game and enjoy a collaborative experience by operating the crane in real time.

[0169] Prompt Sentence Examples

[0170] "After user A customizes his avatar with a blue shirt and jeans and a virtual character with a black cat, explain how that information is shared with other users in real time via a server."

[0171] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0172] Step 1:

[0173] Avatar / Virtual Character Customization

[0174] The terminal provides the user with an interface for customizing the avatar and virtual character.

[0175] Input: User selection (clothes, hats, accessories, virtual character type and characteristics)

[0176] Output: Custom data (e.g., JSON format)

[0177] The user uses the provided interface to make selections to customize their avatar or virtual character, such as choosing a blue shirt or a black cat.

[0178] This generates customization data within the device. Specifically, the user's selections are constructed as JSON format data.

[0179] Step 2:

[0180] Data storage and sharing

[0181] The terminal transmits the generated customization data to the server.

[0182] Input: Customization data

[0183] Output: Data sent to the server

[0184] The server stores the received customization data in a database.

[0185] Input: Customization data

[0186] Output: Records in the database

[0187] The server distributes the customized data to other users in real time using the data sharing means.

[0188] Input: Saved customization data

[0189] Output: Data distributed to other users

[0190] The other user's device will then render the avatar or virtual character based on the received customization data. Specifically, it will analyze the JSON data and render it in real time using OpenGL, DirectX, etc.

[0191] Step 3:

[0192] Obtaining viewpoint information

[0193] The device acquires the user's viewpoint information using the built-in camera and sensors.

[0194] Input: User's viewpoint movement

[0195] Output: Viewpoint information (position, direction, gaze point)

[0196] The device accurately measures the user's location and gaze direction, and collects this data in real time. Specifically, it combines data from the IMU and GPS to form viewpoint information.

[0197] Step 4:

[0198] Sending viewpoint information

[0199] The terminal transmits the acquired viewpoint information to the server.

[0200] Input: Viewpoint information

[0201] Output: Viewpoint information sent to the server

[0202] The viewpoint information is sent to the server using a high-speed communication protocol (e.g., WebSocket) with minimal latency.

[0203] Step 5:

[0204] Real-time drawing

[0205] The server uses a generation AI based on the received viewpoint information to generate instructions for drawing virtual characters and avatars in real time.

[0206] Input: Viewpoint information, customization data

[0207] Output: Drawing instructions

[0208] Using a generative AI model (e.g., GAN), data for real-time drawing is generated based on the user's viewpoint information and customization data.

[0209] The server transmits the generated drawing instructions to the terminal.

[0210] Input: Drawing instructions

[0211] Output: Drawing instructions sent to the device

[0212] Step 6:

[0213] Displaying avatars and virtual characters

[0214] The terminal displays an avatar or virtual character in the user's field of view based on the received drawing instructions.

[0215] Input: Drawing instructions

[0216] Output: Avatars and virtual characters displayed on the screen

[0217] Use a rendering engine (e.g. Unity, Unreal Engine) to visualize the drawing instructions.

[0218] Step 7:

[0219] Providing a shared experience

[0220] A server distributes information about augmented reality content installed in a specific location.

[0221] Input: Augmented reality content information

[0222] Output: Augmented reality content information delivered to the device

[0223] The terminal renders the virtual content at that location based on the received augmented reality content information.

[0224] Input: Augmented reality content information

[0225] Output: Virtual content displayed on a display

[0226] Users can interact with AR content and share the experience with other users in real time.

[0227] Input: User operation information

[0228] Output: Realizing a collaborative experience

[0229] User operation information is shared with other users via the server, enabling an interactive experience.

[0230] (Application example 1)

[0231] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0232] In modern brick-and-mortar stores, it is difficult for customers to determine which product is most suitable for them when selecting a product. Furthermore, there is a lack of means for real-time interaction when multiple customers are selecting and trying out products together. This results in a poor shopping experience and often leads to suboptimal purchasing decisions. Furthermore, the lack of tools to visually check product details and discount information in real time can discourage purchases.

[0233] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0234] In this invention, the server includes: a customization unit that enables a user to customize their own avatar and virtual pet; a storage unit that stores the customization data generated by the customization unit; a sharing unit that shares the customization data stored in the storage unit with other users; an acquisition unit that acquires user viewpoint information; a drawing unit that uses a generative AI model to draw the virtual pet and avatar based on the viewpoint information acquired by the acquisition unit; a display unit that displays the avatar and virtual pet drawn by the drawing unit in the user's field of view; an information display unit that acquires product information from a physical store and displays it in the user's field of view in real time; and a collaborative experience unit that enables multiple users to try on products together and interact in real time. This allows customers to enjoy a collaborative shopping experience with other customers while visually checking product information and discount information in real time.

[0235] An "avatar" is a digital character or icon that a user uses to represent themselves in a virtual environment.

[0236] A "virtual pet" is a digitally generated animal or character that a user can keep within a virtual environment.

[0237] "Customization means" refers to functions or modules that allow users to change the appearance or attributes of their avatar or virtual pet.

[0238] "Storage" refers to a data storage system for temporarily or permanently storing the generated customization data.

[0239] The "sharing means" is a function or module that transmits the saved customized data to other users and shares it in real time.

[0240] The "acquisition means" is an input device such as a camera or sensor for acquiring user viewpoint information.

[0241] A "generative AI model" is an artificial intelligence algorithm that draws avatars and virtual pets in real time based on the user's viewpoint information and customization data.

[0242] A "rendering tool" is a graphics engine or software that displays an avatar or virtual pet based on data obtained using a generative AI model.

[0243] The "display means" refers to a display device or AR device that displays the rendered avatar or virtual pet in the user's field of view.

[0244] "Information display means" refers to a function or module that acquires product information and discount information from a physical store in real time and displays it in the user's field of view.

[0245] "Collaborative experience means" refers to functions and systems that allow multiple users to try on products simultaneously and interact in real time.

[0246] This invention provides a system that allows users to customize their own avatars and virtual pets and share and interact with other users in real time to improve the customer shopping experience in physical stores. Specific embodiments of this system are described below.

[0247] Hardware and software used

[0248] Smartphones and smart glasses: Capture user perspective information and use it to display customized data.

[0249] Cameras and sensors: Input devices for acquiring user viewpoint information.

[0250] Database Management: Use Amazon RDS (relational data store) to store customized data.

[0251] Viewpoint information processing: We use OpenCV (an open-source computer vision library) to process viewpoint information.

[0252] Generative AI model: Uses Google Cloud AI (generative AI service) to render avatars and virtual pets in real time.

[0253] Real-time communication: Use Firebase (a real-time database) to share data in real time.

[0254] Specific operation of the system

[0255] Customize your avatar and virtual pet

[0256] Users use their smartphones or smart glasses to customize their avatars and virtual pets, including clothing, hats, accessories, and the type and characteristics of their virtual pets. Once customization is complete, the data is stored on a server.

[0257] Data storage and sharing

[0258] The server stores the generated customization data in Amazon RDS, and then shares the data in real time with other users in the same physical store using Firebase.

[0259] Obtaining viewpoint information and drawing in real time

[0260] The cameras and sensors in smartphones and smart glasses capture the user's viewpoint. This information is sent to a server and processed by OpenCV. The server uses a generative AI model to generate real-time instructions for rendering an avatar or virtual pet based on the viewpoint and sends them to each device.

[0261] Displaying product information

[0262] When a user points the camera at a product using a smartphone or smart glasses, the server obtains product information and discount information and displays it in the user's field of view in real time.

[0263] Providing a shared experience

[0264] The server provides functionality for multiple users to collaboratively try on the same product and interact in real time, allowing users to collaboratively select products and make purchasing decisions with other users.

[0265] Specific examples

[0266] Example 1: User A and User B are in the same store

[0267] First, User A customizes his or her avatar with a blue shirt and jeans, and his or her virtual pet with a black cat. Once customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to User B's device. User B's device then renders User A's avatar and the black cat in real time.

[0268] Example 2: Virtual try-on session in a physical store

[0269] The server acquires product information from the physical store and distributes related information to all devices. The devices receive the information and display a virtual try-on opportunity within the user's field of view. User A and User B can virtually try on the same item and check the results in real time, allowing for a collaborative experience.

[0270] Prompt sentence input example

[0271] "Prompt for real-time rendering of avatars and virtual pets from user viewpoint information"

[0272] This will improve the in-store shopping experience and allow customers to make more effective product selections.

[0273] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0274] Step 1:

[0275] A user customizes an avatar and a virtual pet using a smartphone or smart glasses, selecting clothes, hats, accessories, and the type and characteristics of the virtual pet to generate customization data.

[0276] Input: User customization choices (clothes, hats, accessories, virtual pets)

[0277] Output: Customization data (avatar and virtual pet information)

[0278] Step 2:

[0279] The device sends the generated customization data to the server, which stores it in Amazon RDS.

[0280] Input: Customization data

[0281] Output: Saved customization data

[0282] Step 3:

[0283] The server shares the saved customization data in real time with other users in the same physical store, sending the data in real time using Firebase.

[0284] Input: Saved customization data

[0285] Output: Shared customization data

[0286] Step 4:

[0287] The camera and sensors on the device capture the user's viewpoint, which is then sent to the server.

[0288] Input: User's viewpoint information (camera, sensor)

[0289] Output: Viewpoint information sent to the server

[0290] Step 5:

[0291] The server processes the viewpoint information using OpenCV and uses a generative AI model to generate drawing instructions for the avatar and virtual pet based on the viewpoint information.

[0292] Input: Viewpoint information

[0293] Output: Drawing instructions

[0294] Step 6:

[0295] The server generates and sends drawing instructions to each device, which then draws the avatar and virtual pet in real time based on the drawing instructions.

[0296] Input: Drawing instructions

[0297] Output: Real-time rendered avatars and virtual pets

[0298] Step 7:

[0299] The device displays the rendered avatar and virtual pet in the user's field of view, using a display device such as a smartphone or smart glasses.

[0300] Input: Real-time rendered avatars and virtual pets

[0301] Output: Avatar and virtual pet displayed in the user's field of view

[0302] Step 8:

[0303] When a user points the camera at a product in a physical store, the device acquires product information, which is then sent to a server, which then acquires product information and discount information.

[0304] Input: Product information (input from camera)

[0305] Output: Retrieved product information and discount information

[0306] Step 9:

[0307] The server transmits the acquired product information and discount information to the terminal in real time, and the terminal displays the information in the user's field of view.

[0308] Input: Product information and discount information

[0309] Output: Product information and discount information displayed in the user's field of view

[0310] Step 10:

[0311] The server sends information to the devices so that multiple users can try on products simultaneously and interact with each other in real time. Each device displays the try-on information and interaction information in the user's field of view, enabling a collaborative experience.

[0312] Input: Try-on information and interaction information

[0313] Output: Try-on information and interaction information displayed in the user's field of view

[0314] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0315] The purpose of the experience-sharing AR system of the present invention is to provide a more immersive collaborative experience by allowing users to customize their own avatars and virtual pets, share them with other users in real time, and reflect the users' emotions using an emotion engine. An embodiment of this system is described below in detail.

[0316] overview

[0317] The system includes a means for users to customize their own avatars and virtual pets, and a data sharing means for sharing them with other users. It also uses a generation AI to acquire user viewpoint information and, based on that information, render the virtual pets and avatars in real time, displaying them in the user's field of view. It also incorporates an emotion engine that recognizes the user's emotions, and reflects the recognized emotion information in the behavior and appearance of the avatars and virtual pets, providing a more realistic experience.

[0318] Program processing

[0319] 1. Customize your avatar and virtual pet

[0320] The terminal provides the user with an interface for customizing their avatar and virtual pet.

[0321] The user selects the clothes, hats, accessories, type and characteristics of the virtual pet.

[0322] The terminal generates customization data and transmits it to the server.

[0323] 2. Data storage and sharing

[0324] The server stores the received customization data in a database.

[0325] The server distributes the customization data to other users wearing the same AR device.

[0326] The other user's device renders the avatar or virtual pet based on the received customization data.

[0327] 3. Real-time drawing

[0328] The device acquires the user's viewpoint information using a camera or sensor.

[0329] The terminal transmits viewpoint information to the server.

[0330] The server uses a generative AI based on the received viewpoint information to create instructions for drawing a virtual pet or avatar in real time.

[0331] The server sends drawing instructions to the terminal.

[0332] The terminal displays the avatar or virtual pet in the user's field of view based on the received drawing instructions.

[0333] 4. Introducing the Emotion Engine

[0334] The terminal recognizes emotions by analyzing the user's facial expressions, voice, and behavioral patterns.

[0335] The emotion engine sends the recognized emotion information to the server.

[0336] The server generates instructions to change the behavior and appearance of the avatar and virtual pet in real time based on the emotional information.

[0337] The user's emotional information is shared with other users and reflected in the avatars and virtual pets of the other users.

[0338] 5. Providing a collaborative experience

[0339] The server distributes information about AR content installed in a specific location (e.g., advertising space on a train).

[0340] The device draws virtual content (e.g., a crane game) at that location based on the received AR content information.

[0341] Users can interact with AR content and share the experience with other users in real time.

[0342] Specific examples

[0343] Example 1: User A and User B are on the same train

[0344] First, user A customizes his or her avatar with a blue shirt and jeans, and customizes his or her virtual pet with a black cat. Once the customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to user B's device. User B's device then renders user A's avatar and the black cat in real time. Similarly, if user B changes his or her avatar to a red hat and sets his or her virtual pet to a white dog, the data is shared using the same procedure. Furthermore, the emotion engine recognizes user A's smile and reflects it by making the avatar smile.

[0345] Example 2: Virtual "crane game" on a train

[0346] The server sets up a virtual "crane game" in an advertising space on a train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B participate in the crane game and operate the crane in real time. The emotion engine recognizes User B's excitement, and the avatar's behavior changes to match that emotion.

[0347] In this way, the system of the present invention allows users to share AR content with other users in real time and enjoy a immersive collaborative experience that reflects their emotions.

[0348] The processing flow will be explained below.

[0349] Below, we will explain the processing of the program of the invention, which combines an experience-sharing AR system with an emotion engine, by dividing it into specific steps.

[0350] Customize your avatar and virtual pet

[0351] Step 1:

[0352] The user launches the "Avatar Connect" application.

[0353] The user opens the application and is presented with a login screen.

[0354] Step 2:

[0355] The user enters their login information and logs in.

[0356] The user enters their account information and clicks the "Login" button.

[0357] The application sends the login information to the server.

[0358] Step 3:

[0359] The server authenticates the user's login information.

[0360] The server checks the database to verify the login information.

[0361] The authentication result is returned to the terminal.

[0362] Step 4:

[0363] The terminal displays a customization screen to the user.

[0364] If authentication is successful, the customization interface will be displayed.

[0365] Users can choose the appearance of their avatar and virtual pet.

[0366] Step 5:

[0367] Users customize their avatars and virtual pets.

[0368] Choose from multiple options (clothes, hats, pet types, etc.).

[0369] Confirm your customizations with the "Save" button.

[0370] Step 6:

[0371] The terminal transmits the customization data to the server.

[0372] The terminal generates customization data and transmits it to the server.

[0373] Data storage and sharing

[0374] Step 7:

[0375] The server stores the customization data in a database.

[0376] The received customization data is recorded in a database.

[0377] Step 8:

[0378] The server distributes the data to other users.

[0379] Data is distributed in real time to other users wearing the same AR device.

[0380] Step 9:

[0381] The terminal receives customization information of other users.

[0382] Based on the customization information received by the terminal, the avatars and virtual pets of other users are drawn.

[0383] Real-time drawing

[0384] Step 10:

[0385] The terminal acquires the user's viewpoint information.

[0386] Cameras and sensors are used to obtain the user's viewpoint information (position, direction).

[0387] Step 11:

[0388] The terminal transmits viewpoint information to the server.

[0389] The acquired viewpoint information is sent to the server in real time.

[0390] Step 12:

[0391] The server creates drawing instructions using a generation AI.

[0392] Based on the received viewpoint information, the generation AI generates drawing instructions for the virtual pet or avatar.

[0393] Sends drawing instructions to the device.

[0394] Step 13:

[0395] The terminal performs real-time drawing.

[0396] Based on instructions received from the server, an avatar or virtual pet is displayed in the user's field of view.

[0397] Introducing the Emotion Engine

[0398] Step 14:

[0399] The device analyzes the user's facial expressions, voice, and behavioral patterns.

[0400] The built-in camera, microphone, and motion sensor are used to obtain user emotional information.

[0401] Step 15:

[0402] The emotion engine analyzes the recognized emotion information.

[0403] The emotion engine analyzes the acquired information and identifies the user's emotional state.

[0404] Step 16:

[0405] The emotion engine sends the emotion information to the server.

[0406] Emotional information is sent to the server in real time.

[0407] Step 17:

[0408] The server generates instructions to change the behavior and appearance of the avatar and virtual pet based on the emotional information.

[0409] Based on the user's emotional information, instructions are generated to change the behavior and appearance of the avatar or virtual pet in real time.

[0410] Step 18:

[0411] The server shares the emotion information with other users.

[0412] Emotional information is also distributed to other users who share the experience in the same space.

[0413] Step 19:

[0414] The device receives the emotional information and changes the behavior and appearance of the avatar and virtual pet.

[0415] The behavior and appearance of the avatar and virtual pet are changed based on the received emotional information.

[0416] Providing a shared experience

[0417] Step 20:

[0418] The server distributes information about AR content placed in a specific location.

[0419] Virtual content (such as a crane game) is placed in a specific location (e.g., advertising space on a train), and that information is sent to all devices.

[0420] Step 21:

[0421] The terminal receives the content information and renders it.

[0422] Based on the received AR content information, virtual content is drawn at that location.

[0423] Step 22:

[0424] Users can share experiences with AR content.

[0425] Multiple users can simultaneously interact with virtual content and enjoy a collaborative experience in real time.

[0426] Example 2

[0427] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0428] While current AR systems allow users to customize their avatars and virtual pets, they struggle to share them in real time and provide a more immersive experience by incorporating emotions. They also lack a means to effectively use generative AI models to render images in real time based on the user's viewpoint.

[0429] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: customization means for allowing a user to customize his or her own avatar and virtual pet; data storage means for storing customization data generated by the customization means; data sharing means for sharing the customization data stored in the storage means with other users; viewpoint acquisition means for acquiring user viewpoint information; drawing means for drawing the virtual pet and avatar using a generative AI model based on the viewpoint information acquired by the viewpoint acquisition means; display means for displaying the avatar and virtual pet drawn by the drawing means in the user's field of view; emotion recognition means for analyzing the user's facial expressions, voice, and behavior patterns to recognize emotions; and emotion reflection means for changing the behavior and appearance of the avatar and virtual pet in real time based on the user's emotion recognized by the emotion recognition means. This allows users to share their avatars and virtual pets with other users in real time, providing a more realistic experience with emotions reflected.

[0430] "Customization means" refers to a means by which a user can freely change and set their own avatar and virtual pet.

[0431] The "data storage means" is a means for storing the customization data generated by the customization means.

[0432] The "data sharing means" is a means for sharing the customized data stored in the storage means with other users.

[0433] The "viewpoint acquisition means" is a means for acquiring user viewpoint information.

[0434] A "generative AI model" is an artificial intelligence model that renders virtual pets and avatars in real time based on the user's viewpoint information.

[0435] The "drawing means" is a means for drawing a virtual pet and an avatar using a generating AI model based on the viewpoint information acquired by the viewpoint acquisition means.

[0436] The "display means" is a means for displaying the avatar and virtual pet drawn by the drawing means in the user's field of view.

[0437] The "emotion recognition means" is a means for recognizing emotions by analyzing the user's facial expressions, voice, and behavioral patterns.

[0438] The "emotion reflection means" is a means for changing the behavior and appearance of the avatar and virtual pet in real time based on the user's emotion recognized by the emotion recognition means.

[0439] MODE FOR CARRYING OUT THE INVENTION

[0440] The present invention is an experience-sharing AR system that allows users to customize avatars and virtual pets in real time, share them with other users, and reflect emotions to provide an immersive experience. The system includes the following main means:

[0441] Customization methods

[0442] The device provides the user with an interface for customizing their avatar and virtual pet. The user selects clothing, accessories, and the type of virtual pet, and customization data is generated based on this selection. This data can then be shared with other users via a data sharing mechanism.

[0443] Data storage means

[0444] The server stores the customization data received from the terminal in a database, which centrally manages the customization information for each user.

[0445] Data sharing methods

[0446] The server distributes the stored customization data in real time to other users wearing augmented reality devices in the same space. The other users' devices then render avatars and virtual pets based on the received customization data.

[0447] Perspective acquisition method

[0448] The device captures the user's viewpoint using a camera or sensor, which is then sent to a server where it is used for real-time rendering using a generative AI model.

[0449] Generative AI model and rendering method

[0450] The server uses the generative AI model based on the viewpoint information acquired by the viewpoint acquisition means to create instructions for drawing the virtual pet and avatar, which then causes the virtual pet and avatar to be displayed in the user's field of view in real time.

[0451] Display means

[0452] Based on the drawing instructions received by the device, the avatar or virtual pet is displayed within the user's field of view, allowing the user to see their customized character in the AR space.

[0453] Emotion recognition and reflection measures

[0454] The device analyzes the user's facial expressions, voice, and behavioral patterns to recognize emotions. The emotion information obtained by the emotion recognition means is sent to a server, which generates instructions to change the behavior and appearance of the avatar and virtual pet in real time based on that information. This allows the user's emotions to be reflected in the avatar and virtual pet, providing a more realistic experience.

[0455] Specific examples

[0456] Example 1: User A and User B are on the same train

[0457] First, user A customizes his or her avatar with a blue shirt and jeans, and customizes his or her virtual pet with a black cat. Once customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to user B's device. User B's device then uses the data to draw the avatar and black cat in real time. The emotion engine recognizes user A's smile and reflects it so that the avatar smiles.

[0458] Example 2: Virtual "crane game" on a train

[0459] The server sets up a virtual "crane game" in an advertising space on a train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B participate in the crane game and operate the crane in real time. The emotion engine recognizes User B's excitement, and the avatar's behavior changes to match that emotion.

[0460] Prompt Sentence Examples

[0461] "How would the system behave if a user changed their avatar to a red shirt and blue jeans and their virtual pet to a white dog?"

[0462] "Please explain in detail the steps to install a virtual crane game inside a car."

[0463] In this way, the system of the present invention allows users to customize, share, and even emote avatars and virtual pets in real time, providing an immersive AR experience.

[0464] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0465] Program processing flow and specific explanation

[0466] Step 1: Display the avatar / virtual pet customization interface

[0467] The terminal provides the user with an interface for customizing their avatar and virtual pet.

[0468] Input: Application launch and user selection actions.

[0469] Output: The interface is displayed on the device display.

[0470] What it does: A screen appears on your smartphone or tablet where you can select clothes, accessories, and a virtual pet for your avatar.

[0471] Step 2: User selects options

[0472] The user operates the interface to select clothes, hats, accessories, and the type and characteristics of the virtual pet.

[0473] Input: User taps and swipes.

[0474] Output: Data for the selected option.

[0475] What happens: The user taps to select an option, such as a blue shirt, jeans, or a black cat.

[0476] Step 3: Generate customization data

[0477] The terminal generates customization data based on the selected options.

[0478] Input: Option data selected by the user.

[0479] Output: Customization data (e.g. JSON file).

[0480] Specific behavior: Information about the selected options is compiled into a single JSON file.

[0481] Step 4: Submit customization data

[0482] The terminal transmits the customization data to the server.

[0483] Input: Customization data.

[0484] Output: Sending data to the server.

[0485] What happens: A JSON file is uploaded to a server over the internet.

[0486] Step 5: Save your data

[0487] The server stores the received customization data in a database.

[0488] Input: The submitted customization data.

[0489] Output: Customization data stored in a database.

[0490] What happens: The contents of the JSON file are inserted into a specific table in the database.

[0491] Step 6: Serving Data

[0492] The server distributes the stored customization data in real time to other users wearing augmented reality devices in the same space.

[0493] Input: Customization data stored in the database.

[0494] Output: Data sent to other users' devices.

[0495] Specific operation: Customized data extracted from the database is pushed to other users' devices.

[0496] Step 7: Obtaining viewpoint information

[0497] The device acquires the user's viewpoint information using a camera or sensor.

[0498] Input: Camera video data and sensor location information.

[0499] Output: Viewpoint information.

[0500] Specific operations: The camera captures images, and sensors detect position and tilt.

[0501] Step 8: Sending viewpoint information

[0502] The terminal transmits viewpoint information to the server.

[0503] Input: Obtained viewpoint information.

[0504] Output: Send viewpoint information to the server.

[0505] Specific operation: Location information and image data are uploaded to the server.

[0506] Step 9: Generate drawing instructions

[0507] The server uses a generative AI model based on the viewpoint information to create instructions for drawing the virtual pet and avatar.

[0508] Input: viewpoint information.

[0509] Output: Drawing instruction data.

[0510] Specific operation: The generative AI analyzes viewpoint information and generates new drawing guidance.

[0511] Step 10: Sending drawing instructions

[0512] The server sends drawing instructions to the terminal.

[0513] Input: Drawing instruction data.

[0514] Output: Sending drawing instructions to the device.

[0515] Specific behavior: The newly generated drawing instruction data is pushed to the device.

[0516] Step 11: Drawing within the field of view

[0517] Based on the drawing instructions received by the terminal, an avatar or virtual pet is displayed in the user's field of view.

[0518] Input: Drawing instruction data.

[0519] Output: An avatar or virtual pet displayed in the user's field of view.

[0520] What happens: The device display updates with the new drawing.

[0521] Step 12: Emotion Recognition

[0522] The device analyzes the user's facial expressions, voice, and behavioral patterns to recognize emotions.

[0523] Input: Camera footage, audio data, behavioral data.

[0524] Output: Recognized emotion information.

[0525] How it works: The camera captures facial expressions, the microphone records audio, and this information is analyzed by analytics software.

[0526] Step 13: Sending Emotional Information

[0527] The emotion engine sends the recognized emotion information to the server.

[0528] Input: Emotion information.

[0529] Output: Sending emotion information to the server.

[0530] Specific operation: Emotion information data is uploaded to the server.

[0531] Step 14: Instructions for behavior and appearance changes

[0532] The server generates instructions for changing the behavior and appearance of the avatar and virtual pet in real time based on the emotional information.

[0533] Input: Emotion information.

[0534] Output: Instruction data for changing behavior and appearance.

[0535] Specific behavior: The server generates instructions for behavior and appearance changes based on the new emotion.

[0536] Step 15: Sharing Emotional Information

[0537] The server shares the user's emotional information with other users and reflects it in the avatars and virtual pets of the other users.

[0538] Input: Emotion information.

[0539] Output: Sending emotion information to other users' devices.

[0540] Specific operation: Emotion information is pushed to other users' devices.

[0541] Step 16: Drawing after reflecting emotions

[0542] The device updates the behavior and appearance of the avatar or virtual pet based on the emotional information and redraws it in the user's field of view.

[0543] Input: Data instructing behavior and appearance changes.

[0544] Output: Updated view of your avatar and virtual pet.

[0545] What happens: The device display will update to show the new emotion-based drawing.

[0546] This process allows users to share their customized avatars and virtual pets with other users in real time, creating a more emotionally immersive experience.

[0547] (Application example 2)

[0548] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0549] In today's physical store experience, it is difficult for customers to share emotions or customized characters with other customers in real time, and individual experiences tend to be emphasized. This results in a lack of interaction and collaborative experiences between customers in the store, and a lack of realism and enjoyment. Furthermore, a shopping experience using dynamic avatars or virtual pets that reflect emotions has not yet been realized.

[0550] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0551] In this invention, the server includes customization means for allowing users to customize their own avatars and virtual pets, data storage means for storing customization data generated by the customization means, data sharing means for sharing the customization data stored in the storage means with other users, viewpoint acquisition means for acquiring user viewpoint information, drawing means for using a generation AI to draw the virtual pet and avatar based on the viewpoint information acquired by the viewpoint acquisition means, display means for displaying the avatar and virtual pet drawn by the drawing means in the user's field of view, emotion engine means for detecting the user's emotions and changing the behavior and appearance of the avatar and virtual pet based on the emotions, and emotion sharing means for sharing the emotion information detected by the emotion engine means with other users. This allows users to share their emotions with other customers in real time, providing a dynamic and interactive shopping experience.

[0552] A "customization means" is a method that provides an interface for a user to individually configure their avatar and virtual pet, and select and edit their appearance and behavior.

[0553] "Data storage means" refers to a database or storage system for temporarily or permanently recording user-customized avatar and virtual pet settings.

[0554] The "data sharing means" is a communication means for transmitting the customization data generated by the customization means to other users in real time or with a certain time lag.

[0555] The "viewpoint acquisition means" is a device including a camera or sensor for collecting information about the direction and field of view of the user.

[0556] "Rendering means" refers to a method for displaying virtual pets and avatars superimposed on the real world using a generating AI based on the user's viewpoint information.

[0557] The "display means" is a technology for displaying the avatar and virtual pet generated by the drawing means on the display of the user's device (such as a smartphone or smart glasses).

[0558] The "emotion engine means" is a system that analyzes the user's facial expressions, voice, and behavioral patterns to recognize emotions, and reflects this emotional information in the avatar or virtual pet.

[0559] The "emotion sharing means" is a communication means for transmitting and sharing the user's emotion information recognized by the emotion engine means with other users.

[0560] "Generative AI" is an artificial intelligence technology that generates and renders virtual objects in real time based on user input data and viewpoint information.

[0561] The present invention allows users to customize their own avatars and virtual pets, share them with other users in real time, and use an emotion engine to reflect the user's emotions, providing a more immersive collaborative experience. Specific embodiments are described below.

[0562] overview

[0563] This system allows users to customize avatars and virtual pets using devices such as smartphones or smart glasses, and share them with other users in real time. Furthermore, an emotion engine can recognize users' emotions in real time and reflect them in the avatars and virtual pets.

[0564] Hardware and Software

[0565] Hardware:

[0566] Smartphone, smart glasses, or head-mounted display (e.g. Microsoft HoloLens)

[0567] Cameras and sensors (for obtaining viewpoint information)

[0568] Server (for data storage and processing)

[0569] software:

[0570] OpenCV (image processing library)

[0571] Dlib (Facial Recognition Library)

[0572] EmotionRecognition (emotion recognition engine)

[0573] AvatarCustomization (Avatar Customization Interface)

[0574] ArServerSdk (AR Server SDK)

[0575] Processing flow

[0576] 1. Providing customization methods

[0577] Users use a smartphone or smart glasses interface to customize their avatar and virtual pet, selecting clothing, accessories, and the type and characteristics of their virtual pet.

[0578] 2. Data storage and sharing

[0579] The customized data is sent to the server and stored in a database, and the data is distributed in real time so that other users can share this customized data in the same space.

[0580] 3. Obtaining viewpoint information and drawing

[0581] Cameras and sensors capture the user's viewpoint information and send it to a server. The AI ​​generates instructions for drawing the virtual pet and avatar based on the viewpoint information and sends them to the user's device. The device then draws the images in real time based on these instructions.

[0582] 4. Introducing the Emotion Engine

[0583] The system recognizes emotions from the user's facial expressions and voice and sends them to the server, which uses this information to generate instructions to change the behavior and appearance of the avatar and virtual pet. Emotional information is also shared with other users.

[0584] Specific examples

[0585] For example, imagine a user puts on smart glasses in a brick-and-mortar store and customizes their avatar with a pink dress, a blue hat, and a brown dog. This information is displayed in real time to other users. When the emotion engine recognizes the user's smile, the avatar also changes to a smile, and the change is communicated to other users.

[0586] Prompt Sentence Examples

[0587] "The avatar's outfit is a pink dress, the hat is blue, and the pet is a brown dog. If the user is smiling, the avatar should also smile."

[0588] In this way, the system of the present invention allows users to customize their avatars and virtual pets, sharing emotions with other users in real time, providing an immersive, interactive, and collaborative experience.

[0589] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0590] Step 1:

[0591] The terminal provides a customization interface for users to customize their avatars and virtual pets.

[0592] This includes an interface where the user can select the avatar's clothing, accessories, and the type and characteristics of the virtual pet, and the device generates this as customization data as the user provides the input.

[0593] Input: User selection information (clothing, accessories, pet characteristics)

[0594] Output: Customization data

[0595] Step 2:

[0596] The terminal transmits the generated customization data to the server.

[0597] The server stores the received customization data in a data storage means, so that the customization information is safely recorded and can be reused later.

[0598] Input: Customization data

[0599] Output: Customization data stored in the database

[0600] Step 3:

[0601] The server distributes the saved customization data in real time to other users wearing AR devices in the same space.

[0602] The other users' terminals prepare to render the avatars and virtual pets based on the received customization data.

[0603] Input: Customization data stored in the database

[0604] Output: Customization data distributed to other users' devices

[0605] Step 4:

[0606] The device acquires the user's viewpoint information using a camera or sensor.

[0607] This viewpoint information is sent to the server and used as the basic data for drawing. Based on the user's viewpoint information, the generative AI model creates drawing instructions for the virtual pet and avatar.

[0608] Input: User's viewpoint information (camera image, gaze data)

[0609] Output: Viewpoint information sent to the server

[0610] Step 5:

[0611] The server uses a generative AI model based on the user's viewpoint to generate instructions for rendering the virtual pet and avatar in real time.

[0612] This instruction is sent to the terminal, which displays the avatar and virtual pet in the user's field of view.

[0613] Input: Viewpoint information

[0614] Output: Drawing instructions generated by the generative AI model

[0615] Step 6:

[0616] The device collects data using a camera and microphone to analyze the user's facial expressions and voice.

[0617] The collected data is sent to the emotion engine to recognize the user's emotions, and the recognition results are sent to the server.

[0618] Input: User's facial expression data, voice data

[0619] Output: Recognized emotion data

[0620] Step 7:

[0621] The server generates instructions to change the behavior and appearance of the avatar and the virtual pet based on the emotion information recognized by the emotion engine means.

[0622] These instructions are sent to the device and reflected in the appearance of the avatar and virtual pet in real time.

[0623] Input: Emotion data

[0624] Output: Movement and appearance change instructions

[0625] Step 8:

[0626] The server also shares the recognized emotion information with the terminals of other users.

[0627] Other users can observe in real time how the behavior and appearance of their avatar or virtual pet changes based on the shared emotional information.

[0628] Input: Emotion data

[0629] Output: Emotion data shared with other users' devices

[0630] Specific examples

[0631] This is the process of a user putting on smart glasses in a physical store and customizing their avatar with a pink dress, a blue hat, and a brown dog. The customized information is displayed in real time to other users. When the emotion engine recognizes the user's smile, the avatar also changes to a smile, and the change is communicated to other users.

[0632] An example of a prompt sentence is "The avatar's outfit is a pink dress, the hat is blue, and the pet is a brown dog. When the user smiles, make the avatar smile too."

[0633] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0634] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0635] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0636] [Second embodiment]

[0637] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0638] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0639] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0640] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0641] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0642] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0643] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0644] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0645] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0646] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0647] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0648] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0649] The purpose of the experience sharing AR system of the present invention is to enable multiple users to share the same AR content in real time and interact with each other. An embodiment of this system will be described in detail below.

[0650] overview

[0651] The AR device (terminal) worn by the user has the means to customize the user's avatar and virtual pet and share it with other users, and uses generation AI to render the virtual pet and avatar in real time based on the user's viewpoint, providing the user with an immersive experience.

[0652] Program processing

[0653] 1. Customize your avatar and virtual pet

[0654] The terminal provides the user with an interface for customizing their avatar and virtual pet.

[0655] The user selects the clothes, hats, accessories, type and characteristics of the virtual pet.

[0656] Customized data is generated within the device.

[0657] The terminal transmits the generated customization data to the server.

[0658] 2. Data storage and sharing

[0659] The server stores the received customization data in a database.

[0660] The server distributes the customization data to other users wearing the same AR device.

[0661] The other user's device renders the user's avatar and virtual pet based on the received customization data.

[0662] 3. Real-time drawing

[0663] The device acquires the user's viewpoint information using a camera or sensor.

[0664] The terminal transmits the acquired viewpoint information to the server.

[0665] The server uses a generative AI based on the received viewpoint information to create instructions for drawing a virtual pet or avatar in real time.

[0666] The server sends drawing instructions to the terminal.

[0667] The terminal displays the avatar or virtual pet in the user's field of view based on the received drawing instructions.

[0668] 4. Providing a collaborative experience

[0669] The server distributes information about AR content installed in a specific location (e.g., advertising space on a train).

[0670] The device draws virtual content (e.g., a crane game) at that location based on the received AR content information.

[0671] Users can interact with AR content and share the experience with other users in real time.

[0672] Specific examples

[0673] Example 1: User A and User B are on the same train

[0674] First, user A customizes his or her avatar with a blue shirt and jeans and a virtual pet with a black cat. Once the customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to user B's device. User B's device then renders user A's avatar and the black cat in real time. Similarly, if user B changes his or her avatar to a red hat and sets his or her virtual pet to a white dog, the data is shared using the same procedure.

[0675] Example 2: Virtual "crane game" on a train

[0676] The server sets up a virtual "crane game" in the advertising space on the train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B can participate in the same game and enjoy a collaborative experience by operating the crane in real time.

[0677] In this way, the system of the present invention allows users to share AR content with other users in real time and enjoy an immersive collaborative experience.

[0678] The processing flow will be explained below.

[0679] Below, the program processing of the experience-sharing AR system is explained in concrete steps.

[0680] Customize your avatar and virtual pet

[0681] Step 1:

[0682] The user launches the "Avatar Connect" application.

[0683] The user opens the application and is presented with a login screen.

[0684] Step 2:

[0685] The user enters their login information and logs in.

[0686] The user enters their account information and clicks the "Login" button.

[0687] The application sends the login information to the server.

[0688] Step 3:

[0689] The server authenticates the user's login information.

[0690] The server checks the database to verify the login information.

[0691] The authentication result is returned to the terminal.

[0692] Step 4:

[0693] The terminal displays a customization screen to the user.

[0694] If authentication is successful, the customization interface will be displayed.

[0695] Users can choose the appearance of their avatar and virtual pet.

[0696] Step 5:

[0697] Users customize their avatars and virtual pets.

[0698] Choose from multiple options (clothes, hats, pet types, etc.).

[0699] Confirm your customizations with the "Save" button.

[0700] Step 6:

[0701] The terminal transmits the customization data to the server.

[0702] The terminal generates customization data and transmits it to the server.

[0703] Data storage and sharing

[0704] Step 7:

[0705] The server stores the customization data in a database.

[0706] The received customization data is recorded in a database.

[0707] Step 8:

[0708] The server distributes the data to other users.

[0709] Data is distributed in real time to other users wearing the same AR device.

[0710] Step 9:

[0711] The terminal receives customization information of other users.

[0712] Based on the customization information received by the terminal, the avatars and virtual pets of other users are drawn.

[0713] Real-time drawing

[0714] Step 10:

[0715] The terminal acquires the user's viewpoint information.

[0716] Cameras and sensors are used to obtain the user's viewpoint information (position, direction).

[0717] Step 11:

[0718] The terminal transmits viewpoint information to the server.

[0719] The acquired viewpoint information is sent to the server in real time.

[0720] Step 12:

[0721] The server creates drawing instructions using a generation AI.

[0722] Based on the received viewpoint information, the generation AI generates drawing instructions for the virtual pet or avatar.

[0723] Sends drawing instructions to the device.

[0724] Step 13:

[0725] The terminal performs real-time drawing.

[0726] Based on instructions received from the server, an avatar or virtual pet is displayed in the user's field of view.

[0727] Providing a shared experience

[0728] Step 14:

[0729] The server distributes information about AR content placed in a specific location.

[0730] Virtual content (such as a crane game) is placed in advertising space in a specific location (e.g., inside a train), and that information is sent to all devices.

[0731] Step 15:

[0732] The terminal receives the content information and renders it.

[0733] Based on the received AR content information, virtual content is drawn at that location.

[0734] Step 16:

[0735] Users can share experiences with AR content.

[0736] Multiple users can simultaneously interact with virtual content and enjoy a collaborative experience in real time.

[0737] Example 1

[0738] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0739] In current augmented reality (AR) technology, it has been difficult for users to share their avatars or virtual characters with other users in real time, and even to share experiences collaboratively. Furthermore, there are insufficient means for users to instantly reflect their customized avatars or virtual characters on other users' screens. As a result, the immersive and interactive nature of the AR experience is diminished, resulting in a decrease in user satisfaction.

[0740] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0741] In this invention, the server includes a customization means for allowing a user to customize his or her own avatar and virtual character, a data storage means for storing customization data generated by the customization means, a data sharing means for sharing the customization data stored in the storage means with other users, a viewpoint acquisition means for acquiring user viewpoint information, a drawing means for drawing a virtual character and an avatar using a generating artificial intelligence based on the viewpoint information acquired by the viewpoint acquisition means, a display means for displaying the avatar and virtual character drawn by the drawing means in the user's field of view, and a collaborative experience means for allowing multiple users to share the same augmented reality content in real time and have a collaborative experience. This allows users to experience their own and other users' customized avatars and virtual characters in real time and enjoy a collaborative interactive AR experience.

[0742] 1. An "avatar" is a digital character that represents a user in a virtual environment.

[0743] 2. "Virtual Character" means a digital pet or character that a user can customize and control or display within a virtual environment.

[0744] 3. "Customization Method" means the interface and functionality that allows a User to customize and change the type, characteristics, clothing, accessories, etc. of their Avatar or Virtual Character.

[0745] 4. "Data Storage Means" means a storage device such as a database or memory that stores information about a user's customized avatar and virtual character.

[0746] 5. "Data Sharing Method" means the communication method and functionality for sharing saved customized data with other users.

[0747] 6. "Point of view acquisition means" refers to devices and technologies for acquiring user's point of view information using cameras and sensors.

[0748] 7. “Generative AI” refers to artificial intelligence techniques and models for rendering avatars or virtual characters in real time based on a user’s perspective.

[0749] 8. "Rendering means" means a graphics engine or software that displays an avatar or virtual character in the user's field of view based on the acquired viewpoint information and customization data.

[0750] 9. "Display means" means the technology and devices for displaying a rendered avatar or virtual character on the display of an AR device worn by a user.

[0751] 10. "Collaborative experience means" refers to functions and technologies that enable multiple users to share the same augmented reality content in real time and interact with it collaboratively.

[0752] The present invention provides a system for customizing and sharing avatars and virtual characters in real time with other users using augmented reality (AR) devices worn by users, thereby enabling users to share interactive and immersive experiences with other users. Detailed embodiments of the present invention are described below.

[0753] System Configuration

[0754] This system mainly consists of a customization means, a data storage means, a data sharing means, a viewpoint acquisition means, a drawing means, and a display means. The specific processing of each means is as follows.

[0755] Hardware and software used

[0756] Device: AR device worn by the user (e.g., AR glasses, smartphone)

[0757] Server: Cloud server or on-premise server

[0758] Generative AI model: GAN (generative artificial network)

[0759] Database: MySQL, PostgreSQL

[0760] Communication technology: HTTP, WebSocket

[0761] Graphics API: OpenGL, DirectX

[0762] Rendering engine: Unity, Unreal Engine

[0763] Program processing

[0764] Avatar / Virtual Character Customization

[0765] The device provides the user with an interface for customizing their avatar and virtual character. The user uses this interface to select the avatar's clothing and hat, as well as the type and characteristics of the virtual character. The selected customization data is generated within the device and saved in JSON format or similar.

[0766] Data storage and sharing

[0767] The generated customization data is sent from the device to a server. The server stores the received data in a database and shares it with other users in real time using a data sharing means. The other users' devices analyze the received data and draw the avatar and virtual character.

[0768] Obtaining viewpoint information

[0769] The device acquires the user's viewpoint information using the built-in camera and sensors. This information includes the user's position, orientation, and gaze point. The acquired viewpoint information is sent to the server.

[0770] Real-time drawing

[0771] The server uses a generative AI model based on the received viewpoint information to generate instructions for rendering the virtual character and avatar in real time, which are then sent to the device, which then displays the avatar and virtual character in the user's field of view.

[0772] Providing a shared experience

[0773] The server also distributes information about augmented reality content installed in specific locations, allowing multiple users to share the same content, such as a virtual "crane game" installed on a train. Users can manipulate this content and enjoy a collaborative experience with other users in real time.

[0774] Specific examples

[0775] Example 1: User A and User B are in the same location

[0776] User A customizes his avatar with a blue shirt and jeans and a virtual character with a black cat. Once customization is complete, his device sends this information to the server. The server shares the data with other users and distributes it to User B's device. User B's device then renders User A's avatar and the black cat in real time. Similarly, if User B changes his avatar to a red hat and sets his virtual character to a white dog, the data is shared using the same procedure.

[0777] Example 2: Virtual "crane game" on a train

[0778] The server sets up a virtual "crane game" in the advertising space on the train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B can participate in the same game and enjoy a collaborative experience by operating the crane in real time.

[0779] Prompt Sentence Examples

[0780] "After user A customizes his avatar with a blue shirt and jeans and a virtual character with a black cat, explain how that information is shared with other users in real time via a server."

[0781] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0782] Step 1:

[0783] Avatar / Virtual Character Customization

[0784] The terminal provides the user with an interface for customizing the avatar and virtual character.

[0785] Input: User selection (clothes, hats, accessories, virtual character type and characteristics)

[0786] Output: Custom data (e.g., JSON format)

[0787] The user uses the provided interface to make selections to customize their avatar or virtual character, such as choosing a blue shirt or a black cat.

[0788] This generates customization data within the device. Specifically, the user's selections are constructed as JSON format data.

[0789] Step 2:

[0790] Data storage and sharing

[0791] The terminal transmits the generated customization data to the server.

[0792] Input: Customization data

[0793] Output: Data sent to the server

[0794] The server stores the received customization data in a database.

[0795] Input: Customization data

[0796] Output: Records in the database

[0797] The server distributes the customized data to other users in real time using the data sharing means.

[0798] Input: Saved customization data

[0799] Output: Data distributed to other users

[0800] The other user's device will then render the avatar or virtual character based on the received customization data. Specifically, it will analyze the JSON data and render it in real time using OpenGL, DirectX, etc.

[0801] Step 3:

[0802] Obtaining viewpoint information

[0803] The device acquires the user's viewpoint information using the built-in camera and sensors.

[0804] Input: User's viewpoint movement

[0805] Output: Viewpoint information (position, direction, gaze point)

[0806] The device accurately measures the user's location and gaze direction, and collects this data in real time. Specifically, it combines data from the IMU and GPS to form viewpoint information.

[0807] Step 4:

[0808] Sending viewpoint information

[0809] The terminal transmits the acquired viewpoint information to the server.

[0810] Input: Viewpoint information

[0811] Output: Viewpoint information sent to the server

[0812] The viewpoint information is sent to the server using a high-speed communication protocol (e.g., WebSocket) with minimal latency.

[0813] Step 5:

[0814] Real-time drawing

[0815] The server uses a generation AI based on the received viewpoint information to generate instructions for drawing virtual characters and avatars in real time.

[0816] Input: Viewpoint information, customization data

[0817] Output: Drawing instructions

[0818] Using a generative AI model (e.g., GAN), data for real-time drawing is generated based on the user's viewpoint information and customization data.

[0819] The server transmits the generated drawing instructions to the terminal.

[0820] Input: Drawing instructions

[0821] Output: Drawing instructions sent to the device

[0822] Step 6:

[0823] Displaying avatars and virtual characters

[0824] The terminal displays an avatar or virtual character in the user's field of view based on the received drawing instructions.

[0825] Input: Drawing instructions

[0826] Output: Avatars and virtual characters displayed on the screen

[0827] Use a rendering engine (e.g. Unity, Unreal Engine) to visualize the drawing instructions.

[0828] Step 7:

[0829] Providing a shared experience

[0830] A server distributes information about augmented reality content installed in a specific location.

[0831] Input: Augmented reality content information

[0832] Output: Augmented reality content information delivered to the device

[0833] The terminal renders the virtual content at that location based on the received augmented reality content information.

[0834] Input: Augmented reality content information

[0835] Output: Virtual content displayed on a display

[0836] Users can interact with AR content and share the experience with other users in real time.

[0837] Input: User operation information

[0838] Output: Realizing a collaborative experience

[0839] User operation information is shared with other users via the server, enabling an interactive experience.

[0840] (Application example 1)

[0841] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0842] In modern brick-and-mortar stores, it is difficult for customers to determine which product is most suitable for them when selecting a product. Furthermore, there is a lack of means for real-time interaction when multiple customers are selecting and trying out products together. This results in a poor shopping experience and often leads to suboptimal purchasing decisions. Furthermore, the lack of tools to visually check product details and discount information in real time can discourage purchases.

[0843] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0844] In this invention, the server includes: a customization unit that enables a user to customize their own avatar and virtual pet; a storage unit that stores the customization data generated by the customization unit; a sharing unit that shares the customization data stored in the storage unit with other users; an acquisition unit that acquires user viewpoint information; a drawing unit that uses a generative AI model to draw the virtual pet and avatar based on the viewpoint information acquired by the acquisition unit; a display unit that displays the avatar and virtual pet drawn by the drawing unit in the user's field of view; an information display unit that acquires product information from a physical store and displays it in the user's field of view in real time; and a collaborative experience unit that enables multiple users to try on products together and interact in real time. This allows customers to enjoy a collaborative shopping experience with other customers while visually checking product information and discount information in real time.

[0845] An "avatar" is a digital character or icon that a user uses to represent themselves in a virtual environment.

[0846] A "virtual pet" is a digitally generated animal or character that a user can keep within a virtual environment.

[0847] "Customization means" refers to functions or modules that allow users to change the appearance or attributes of their avatar or virtual pet.

[0848] "Storage" refers to a data storage system for temporarily or permanently storing the generated customization data.

[0849] The "sharing means" is a function or module that transmits the saved customized data to other users and shares it in real time.

[0850] The "acquisition means" is an input device such as a camera or sensor for acquiring user viewpoint information.

[0851] A "generative AI model" is an artificial intelligence algorithm that draws avatars and virtual pets in real time based on the user's viewpoint information and customization data.

[0852] A "rendering tool" is a graphics engine or software that displays an avatar or virtual pet based on data obtained using a generative AI model.

[0853] The "display means" refers to a display device or AR device that displays the rendered avatar or virtual pet in the user's field of view.

[0854] "Information display means" refers to a function or module that acquires product information and discount information from a physical store in real time and displays it in the user's field of view.

[0855] "Collaborative experience means" refers to functions and systems that allow multiple users to try on products simultaneously and interact in real time.

[0856] This invention provides a system that allows users to customize their own avatars and virtual pets and share and interact with other users in real time to improve the customer shopping experience in physical stores. Specific embodiments of this system are described below.

[0857] Hardware and software used

[0858] Smartphones and smart glasses: Capture user perspective information and use it to display customized data.

[0859] Cameras and sensors: Input devices for acquiring user viewpoint information.

[0860] Database Management: Use Amazon RDS (relational data store) to store customized data.

[0861] Viewpoint information processing: We use OpenCV (an open-source computer vision library) to process viewpoint information.

[0862] Generative AI model: Uses Google Cloud AI (generative AI service) to render avatars and virtual pets in real time.

[0863] Real-time communication: Use Firebase (a real-time database) to share data in real time.

[0864] Specific operation of the system

[0865] Customize your avatar and virtual pet

[0866] Users use their smartphones or smart glasses to customize their avatars and virtual pets, including clothing, hats, accessories, and the type and characteristics of their virtual pets. Once customization is complete, the data is stored on a server.

[0867] Data storage and sharing

[0868] The server stores the generated customization data in Amazon RDS, and then shares the data in real time with other users in the same physical store using Firebase.

[0869] Obtaining viewpoint information and drawing in real time

[0870] The cameras and sensors in smartphones and smart glasses capture the user's viewpoint. This information is sent to a server and processed by OpenCV. The server uses a generative AI model to generate real-time instructions for rendering an avatar or virtual pet based on the viewpoint and sends them to each device.

[0871] Displaying product information

[0872] When a user points the camera at a product using a smartphone or smart glasses, the server obtains product information and discount information and displays it in the user's field of view in real time.

[0873] Providing a shared experience

[0874] The server provides functionality for multiple users to collaboratively try on the same product and interact in real time, allowing users to collaboratively select products and make purchasing decisions with other users.

[0875] Specific examples

[0876] Example 1: User A and User B are in the same store

[0877] First, User A customizes his or her avatar with a blue shirt and jeans, and his or her virtual pet with a black cat. Once customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to User B's device. User B's device then renders User A's avatar and the black cat in real time.

[0878] Example 2: Virtual try-on session in a physical store

[0879] The server acquires product information from the physical store and distributes related information to all devices. The devices receive the information and display a virtual try-on opportunity within the user's field of view. User A and User B can virtually try on the same item and check the results in real time, allowing for a collaborative experience.

[0880] Prompt sentence input example

[0881] "Prompt for real-time rendering of avatars and virtual pets from user viewpoint information"

[0882] This will improve the in-store shopping experience and allow customers to make more effective product selections.

[0883] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0884] Step 1:

[0885] A user customizes an avatar and a virtual pet using a smartphone or smart glasses, selecting clothes, hats, accessories, and the type and characteristics of the virtual pet to generate customization data.

[0886] Input: User customization choices (clothes, hats, accessories, virtual pets)

[0887] Output: Customization data (avatar and virtual pet information)

[0888] Step 2:

[0889] The device sends the generated customization data to the server, which stores it in Amazon RDS.

[0890] Input: Customization data

[0891] Output: Saved customization data

[0892] Step 3:

[0893] The server shares the saved customization data in real time with other users in the same physical store, sending the data in real time using Firebase.

[0894] Input: Saved customization data

[0895] Output: Shared customization data

[0896] Step 4:

[0897] The camera and sensors on the device capture the user's viewpoint, which is then sent to the server.

[0898] Input: User's viewpoint information (camera, sensor)

[0899] Output: Viewpoint information sent to the server

[0900] Step 5:

[0901] The server processes the viewpoint information using OpenCV and uses a generative AI model to generate drawing instructions for the avatar and virtual pet based on the viewpoint information.

[0902] Input: Viewpoint information

[0903] Output: Drawing instructions

[0904] Step 6:

[0905] The server generates and sends drawing instructions to each device, which then draws the avatar and virtual pet in real time based on the drawing instructions.

[0906] Input: Drawing instructions

[0907] Output: Real-time rendered avatars and virtual pets

[0908] Step 7:

[0909] The device displays the rendered avatar and virtual pet in the user's field of view, using a display device such as a smartphone or smart glasses.

[0910] Input: Real-time rendered avatars and virtual pets

[0911] Output: Avatar and virtual pet displayed in the user's field of view

[0912] Step 8:

[0913] When a user points the camera at a product in a physical store, the device acquires product information, which is then sent to a server, which then acquires product information and discount information.

[0914] Input: Product information (input from camera)

[0915] Output: Retrieved product information and discount information

[0916] Step 9:

[0917] The server transmits the acquired product information and discount information to the terminal in real time, and the terminal displays the information in the user's field of view.

[0918] Input: Product information and discount information

[0919] Output: Product information and discount information displayed in the user's field of view

[0920] Step 10:

[0921] The server sends information to the devices so that multiple users can try on products simultaneously and interact with each other in real time. Each device displays the try-on information and interaction information in the user's field of view, enabling a collaborative experience.

[0922] Input: Try-on information and interaction information

[0923] Output: Try-on information and interaction information displayed in the user's field of view

[0924] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0925] The purpose of the experience-sharing AR system of the present invention is to provide a more immersive collaborative experience by allowing users to customize their own avatars and virtual pets, share them with other users in real time, and reflect the users' emotions using an emotion engine. An embodiment of this system is described below in detail.

[0926] overview

[0927] The system includes a means for users to customize their own avatars and virtual pets, and a data sharing means for sharing them with other users. It also uses a generation AI to acquire user viewpoint information and, based on that information, render the virtual pets and avatars in real time, displaying them in the user's field of view. It also incorporates an emotion engine that recognizes the user's emotions, and reflects the recognized emotion information in the behavior and appearance of the avatars and virtual pets, providing a more realistic experience.

[0928] Program processing

[0929] 1. Customize your avatar and virtual pet

[0930] The terminal provides the user with an interface for customizing their avatar and virtual pet.

[0931] The user selects the clothes, hats, accessories, type and characteristics of the virtual pet.

[0932] The terminal generates customization data and transmits it to the server.

[0933] 2. Data storage and sharing

[0934] The server stores the received customization data in a database.

[0935] The server distributes the customization data to other users wearing the same AR device.

[0936] The other user's device renders the avatar or virtual pet based on the received customization data.

[0937] 3. Real-time drawing

[0938] The device acquires the user's viewpoint information using a camera or sensor.

[0939] The terminal transmits viewpoint information to the server.

[0940] The server uses a generative AI based on the received viewpoint information to create instructions for drawing a virtual pet or avatar in real time.

[0941] The server sends drawing instructions to the terminal.

[0942] The terminal displays the avatar or virtual pet in the user's field of view based on the received drawing instructions.

[0943] 4. Introducing the Emotion Engine

[0944] The terminal recognizes emotions by analyzing the user's facial expressions, voice, and behavioral patterns.

[0945] The emotion engine sends the recognized emotion information to the server.

[0946] The server generates instructions to change the behavior and appearance of the avatar and virtual pet in real time based on the emotional information.

[0947] The user's emotional information is shared with other users and reflected in the avatars and virtual pets of the other users.

[0948] 5. Providing a collaborative experience

[0949] The server distributes information about AR content installed in a specific location (e.g., advertising space on a train).

[0950] The device draws virtual content (e.g., a crane game) at that location based on the received AR content information.

[0951] Users can interact with AR content and share the experience with other users in real time.

[0952] Specific examples

[0953] Example 1: User A and User B are on the same train

[0954] First, user A customizes his or her avatar with a blue shirt and jeans, and customizes his or her virtual pet with a black cat. Once the customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to user B's device. User B's device then renders user A's avatar and the black cat in real time. Similarly, if user B changes his or her avatar to a red hat and sets his or her virtual pet to a white dog, the data is shared using the same procedure. Furthermore, the emotion engine recognizes user A's smile and reflects it by making the avatar smile.

[0955] Example 2: Virtual "crane game" on a train

[0956] The server sets up a virtual "crane game" in an advertising space on a train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B participate in the crane game and operate the crane in real time. The emotion engine recognizes User B's excitement, and the avatar's behavior changes to match that emotion.

[0957] In this way, the system of the present invention allows users to share AR content with other users in real time and enjoy a immersive collaborative experience that reflects their emotions.

[0958] The processing flow will be explained below.

[0959] Below, we will explain the processing of the program of the invention, which combines an experience-sharing AR system with an emotion engine, by dividing it into specific steps.

[0960] Customize your avatar and virtual pet

[0961] Step 1:

[0962] The user launches the "Avatar Connect" application.

[0963] The user opens the application and is presented with a login screen.

[0964] Step 2:

[0965] The user enters their login information and logs in.

[0966] The user enters their account information and clicks the "Login" button.

[0967] The application sends the login information to the server.

[0968] Step 3:

[0969] The server authenticates the user's login information.

[0970] The server checks the database to verify the login information.

[0971] The authentication result is returned to the terminal.

[0972] Step 4:

[0973] The terminal displays a customization screen to the user.

[0974] If authentication is successful, the customization interface will be displayed.

[0975] Users can choose the appearance of their avatar and virtual pet.

[0976] Step 5:

[0977] Users customize their avatars and virtual pets.

[0978] Choose from multiple options (clothes, hats, pet types, etc.).

[0979] Confirm your customizations with the "Save" button.

[0980] Step 6:

[0981] The terminal transmits the customization data to the server.

[0982] The terminal generates customization data and transmits it to the server.

[0983] Data storage and sharing

[0984] Step 7:

[0985] The server stores the customization data in a database.

[0986] The received customization data is recorded in a database.

[0987] Step 8:

[0988] The server distributes the data to other users.

[0989] Data is distributed in real time to other users wearing the same AR device.

[0990] Step 9:

[0991] The terminal receives customization information of other users.

[0992] Based on the customization information received by the terminal, the avatars and virtual pets of other users are drawn.

[0993] Real-time drawing

[0994] Step 10:

[0995] The terminal acquires the user's viewpoint information.

[0996] Cameras and sensors are used to obtain the user's viewpoint information (position, direction).

[0997] Step 11:

[0998] The terminal transmits viewpoint information to the server.

[0999] The acquired viewpoint information is sent to the server in real time.

[1000] Step 12:

[1001] The server creates drawing instructions using a generation AI.

[1002] Based on the received viewpoint information, the generation AI generates drawing instructions for the virtual pet or avatar.

[1003] Sends drawing instructions to the device.

[1004] Step 13:

[1005] The terminal performs real-time drawing.

[1006] Based on instructions received from the server, an avatar or virtual pet is displayed in the user's field of view.

[1007] Introducing the Emotion Engine

[1008] Step 14:

[1009] The device analyzes the user's facial expressions, voice, and behavioral patterns.

[1010] The built-in camera, microphone, and motion sensor are used to obtain user emotional information.

[1011] Step 15:

[1012] The emotion engine analyzes the recognized emotion information.

[1013] The emotion engine analyzes the acquired information and identifies the user's emotional state.

[1014] Step 16:

[1015] The emotion engine sends the emotion information to the server.

[1016] Emotional information is sent to the server in real time.

[1017] Step 17:

[1018] The server generates instructions to change the behavior and appearance of the avatar and virtual pet based on the emotional information.

[1019] Based on the user's emotional information, instructions are generated to change the behavior and appearance of the avatar or virtual pet in real time.

[1020] Step 18:

[1021] The server shares the emotion information with other users.

[1022] Emotional information is also distributed to other users who share the experience in the same space.

[1023] Step 19:

[1024] The device receives the emotional information and changes the behavior and appearance of the avatar and virtual pet.

[1025] The behavior and appearance of the avatar and virtual pet are changed based on the received emotional information.

[1026] Providing a shared experience

[1027] Step 20:

[1028] The server distributes information about AR content placed in a specific location.

[1029] Virtual content (such as a crane game) is placed in a specific location (e.g., advertising space on a train), and that information is sent to all devices.

[1030] Step 21:

[1031] The terminal receives the content information and renders it.

[1032] Based on the received AR content information, virtual content is drawn at that location.

[1033] Step 22:

[1034] Users can share experiences with AR content.

[1035] Multiple users can simultaneously interact with virtual content and enjoy a collaborative experience in real time.

[1036] Example 2

[1037] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1038] While current AR systems allow users to customize their avatars and virtual pets, they struggle to share them in real time and provide a more immersive experience by incorporating emotions. They also lack a means to effectively use generative AI models to render images in real time based on the user's viewpoint.

[1039] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: customization means for allowing a user to customize his or her own avatar and virtual pet; data storage means for storing customization data generated by the customization means; data sharing means for sharing the customization data stored in the storage means with other users; viewpoint acquisition means for acquiring user viewpoint information; drawing means for drawing the virtual pet and avatar using a generative AI model based on the viewpoint information acquired by the viewpoint acquisition means; display means for displaying the avatar and virtual pet drawn by the drawing means in the user's field of view; emotion recognition means for analyzing the user's facial expressions, voice, and behavior patterns to recognize emotions; and emotion reflection means for changing the behavior and appearance of the avatar and virtual pet in real time based on the user's emotion recognized by the emotion recognition means. This allows users to share their avatars and virtual pets with other users in real time, providing a more realistic experience with emotions reflected.

[1040] "Customization means" refers to a means by which a user can freely change and set their own avatar and virtual pet.

[1041] The "data storage means" is a means for storing the customization data generated by the customization means.

[1042] The "data sharing means" is a means for sharing the customized data stored in the storage means with other users.

[1043] The "viewpoint acquisition means" is a means for acquiring user viewpoint information.

[1044] A "generative AI model" is an artificial intelligence model that renders virtual pets and avatars in real time based on the user's viewpoint information.

[1045] The "drawing means" is a means for drawing a virtual pet and an avatar using a generating AI model based on the viewpoint information acquired by the viewpoint acquisition means.

[1046] The "display means" is a means for displaying the avatar and virtual pet drawn by the drawing means in the user's field of view.

[1047] The "emotion recognition means" is a means for recognizing emotions by analyzing the user's facial expressions, voice, and behavioral patterns.

[1048] The "emotion reflection means" is a means for changing the behavior and appearance of the avatar and virtual pet in real time based on the user's emotion recognized by the emotion recognition means.

[1049] MODE FOR CARRYING OUT THE INVENTION

[1050] The present invention is an experience-sharing AR system that allows users to customize avatars and virtual pets in real time, share them with other users, and reflect emotions to provide an immersive experience. The system includes the following main means:

[1051] Customization methods

[1052] The device provides the user with an interface for customizing their avatar and virtual pet. The user selects clothing, accessories, and the type of virtual pet, and customization data is generated based on this selection. This data can then be shared with other users via a data sharing mechanism.

[1053] Data storage means

[1054] The server stores the customization data received from the terminal in a database, which centrally manages the customization information for each user.

[1055] Data sharing methods

[1056] The server distributes the stored customization data in real time to other users wearing augmented reality devices in the same space. The other users' devices then render avatars and virtual pets based on the received customization data.

[1057] Perspective acquisition method

[1058] The device captures the user's viewpoint using a camera or sensor, which is then sent to a server where it is used for real-time rendering using a generative AI model.

[1059] Generative AI model and rendering method

[1060] The server uses the generative AI model based on the viewpoint information acquired by the viewpoint acquisition means to create instructions for drawing the virtual pet and avatar, which then causes the virtual pet and avatar to be displayed in the user's field of view in real time.

[1061] Display means

[1062] Based on the drawing instructions received by the device, the avatar or virtual pet is displayed within the user's field of view, allowing the user to see their customized character in the AR space.

[1063] Emotion recognition and reflection measures

[1064] The device analyzes the user's facial expressions, voice, and behavioral patterns to recognize emotions. The emotion information obtained by the emotion recognition means is sent to a server, which generates instructions to change the behavior and appearance of the avatar and virtual pet in real time based on that information. This allows the user's emotions to be reflected in the avatar and virtual pet, providing a more realistic experience.

[1065] Specific examples

[1066] Example 1: User A and User B are on the same train

[1067] First, user A customizes his or her avatar with a blue shirt and jeans, and customizes his or her virtual pet with a black cat. Once customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to user B's device. User B's device then uses the data to draw the avatar and black cat in real time. The emotion engine recognizes user A's smile and reflects it so that the avatar smiles.

[1068] Example 2: Virtual "crane game" on a train

[1069] The server sets up a virtual "crane game" in an advertising space on a train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B participate in the crane game and operate the crane in real time. The emotion engine recognizes User B's excitement, and the avatar's behavior changes to match that emotion.

[1070] Prompt Sentence Examples

[1071] "How would the system behave if a user changed their avatar to a red shirt and blue jeans and their virtual pet to a white dog?"

[1072] "Please explain in detail the steps to install a virtual crane game inside a car."

[1073] In this way, the system of the present invention allows users to customize, share, and even emote avatars and virtual pets in real time, providing an immersive AR experience.

[1074] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1075] Program processing flow and specific explanation

[1076] Step 1: Display the avatar / virtual pet customization interface

[1077] The terminal provides the user with an interface for customizing their avatar and virtual pet.

[1078] Input: Application launch and user selection actions.

[1079] Output: The interface is displayed on the device display.

[1080] What it does: A screen appears on your smartphone or tablet where you can select clothes, accessories, and a virtual pet for your avatar.

[1081] Step 2: User selects options

[1082] The user operates the interface to select clothes, hats, accessories, and the type and characteristics of the virtual pet.

[1083] Input: User taps and swipes.

[1084] Output: Data for the selected option.

[1085] What happens: The user taps to select an option, such as a blue shirt, jeans, or a black cat.

[1086] Step 3: Generate customization data

[1087] The terminal generates customization data based on the selected options.

[1088] Input: Option data selected by the user.

[1089] Output: Customization data (e.g. JSON file).

[1090] Specific behavior: Information about the selected options is compiled into a single JSON file.

[1091] Step 4: Submit customization data

[1092] The terminal transmits the customization data to the server.

[1093] Input: Customization data.

[1094] Output: Sending data to the server.

[1095] What happens: A JSON file is uploaded to a server over the internet.

[1096] Step 5: Save your data

[1097] The server stores the received customization data in a database.

[1098] Input: The submitted customization data.

[1099] Output: Customization data stored in a database.

[1100] What happens: The contents of the JSON file are inserted into a specific table in the database.

[1101] Step 6: Serving Data

[1102] The server distributes the stored customization data in real time to other users wearing augmented reality devices in the same space.

[1103] Input: Customization data stored in the database.

[1104] Output: Data sent to other users' devices.

[1105] Specific operation: Customized data extracted from the database is pushed to other users' devices.

[1106] Step 7: Obtaining viewpoint information

[1107] The device acquires the user's viewpoint information using a camera or sensor.

[1108] Input: Camera video data and sensor location information.

[1109] Output: Viewpoint information.

[1110] Specific operations: The camera captures images, and sensors detect position and tilt.

[1111] Step 8: Sending viewpoint information

[1112] The terminal transmits viewpoint information to the server.

[1113] Input: Obtained viewpoint information.

[1114] Output: Send viewpoint information to the server.

[1115] Specific operation: Location information and image data are uploaded to the server.

[1116] Step 9: Generate drawing instructions

[1117] The server uses a generative AI model based on the viewpoint information to create instructions for drawing the virtual pet and avatar.

[1118] Input: viewpoint information.

[1119] Output: Drawing instruction data.

[1120] Specific operation: The generative AI analyzes viewpoint information and generates new drawing guidance.

[1121] Step 10: Sending drawing instructions

[1122] The server sends drawing instructions to the terminal.

[1123] Input: Drawing instruction data.

[1124] Output: Sending drawing instructions to the device.

[1125] Specific behavior: The newly generated drawing instruction data is pushed to the device.

[1126] Step 11: Drawing within the field of view

[1127] Based on the drawing instructions received by the terminal, an avatar or virtual pet is displayed in the user's field of view.

[1128] Input: Drawing instruction data.

[1129] Output: An avatar or virtual pet displayed in the user's field of view.

[1130] What happens: The device display updates with the new drawing.

[1131] Step 12: Emotion Recognition

[1132] The device analyzes the user's facial expressions, voice, and behavioral patterns to recognize emotions.

[1133] Input: Camera footage, audio data, behavioral data.

[1134] Output: Recognized emotion information.

[1135] How it works: The camera captures facial expressions, the microphone records audio, and this information is analyzed by analytics software.

[1136] Step 13: Sending Emotional Information

[1137] The emotion engine sends the recognized emotion information to the server.

[1138] Input: Emotion information.

[1139] Output: Sending emotion information to the server.

[1140] Specific operation: Emotion information data is uploaded to the server.

[1141] Step 14: Instructions for behavior and appearance changes

[1142] The server generates instructions for changing the behavior and appearance of the avatar and virtual pet in real time based on the emotional information.

[1143] Input: Emotion information.

[1144] Output: Instruction data for changing behavior and appearance.

[1145] Specific behavior: The server generates instructions for behavior and appearance changes based on the new emotion.

[1146] Step 15: Sharing Emotional Information

[1147] The server shares the user's emotional information with other users and reflects it in the avatars and virtual pets of the other users.

[1148] Input: Emotion information.

[1149] Output: Sending emotion information to other users' devices.

[1150] Specific operation: Emotion information is pushed to other users' devices.

[1151] Step 16: Drawing after reflecting emotions

[1152] The device updates the behavior and appearance of the avatar or virtual pet based on the emotional information and redraws it in the user's field of view.

[1153] Input: Data instructing behavior and appearance changes.

[1154] Output: Updated view of your avatar and virtual pet.

[1155] What happens: The device display will update to show the new emotion-based drawing.

[1156] This process allows users to share their customized avatars and virtual pets with other users in real time, creating a more emotionally immersive experience.

[1157] (Application example 2)

[1158] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1159] In today's physical store experience, it is difficult for customers to share emotions or customized characters with other customers in real time, and individual experiences tend to be emphasized. This results in a lack of interaction and collaborative experiences between customers in the store, and a lack of realism and enjoyment. Furthermore, a shopping experience using dynamic avatars or virtual pets that reflect emotions has not yet been realized.

[1160] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1161] In this invention, the server includes customization means for allowing users to customize their own avatars and virtual pets, data storage means for storing customization data generated by the customization means, data sharing means for sharing the customization data stored in the storage means with other users, viewpoint acquisition means for acquiring user viewpoint information, drawing means for using a generation AI to draw the virtual pet and avatar based on the viewpoint information acquired by the viewpoint acquisition means, display means for displaying the avatar and virtual pet drawn by the drawing means in the user's field of view, emotion engine means for detecting the user's emotions and changing the behavior and appearance of the avatar and virtual pet based on the emotions, and emotion sharing means for sharing the emotion information detected by the emotion engine means with other users. This allows users to share their emotions with other customers in real time, providing a dynamic and interactive shopping experience.

[1162] A "customization means" is a method that provides an interface for a user to individually configure their avatar and virtual pet, and select and edit their appearance and behavior.

[1163] "Data storage means" refers to a database or storage system for temporarily or permanently recording user-customized avatar and virtual pet settings.

[1164] The "data sharing means" is a communication means for transmitting the customization data generated by the customization means to other users in real time or with a certain time lag.

[1165] The "viewpoint acquisition means" is a device including a camera or sensor for collecting information about the direction and field of view of the user.

[1166] "Rendering means" refers to a method for displaying virtual pets and avatars superimposed on the real world using a generating AI based on the user's viewpoint information.

[1167] The "display means" is a technology for displaying the avatar and virtual pet generated by the drawing means on the display of the user's device (such as a smartphone or smart glasses).

[1168] The "emotion engine means" is a system that analyzes the user's facial expressions, voice, and behavioral patterns to recognize emotions, and reflects this emotional information in the avatar or virtual pet.

[1169] The "emotion sharing means" is a communication means for transmitting and sharing the user's emotion information recognized by the emotion engine means with other users.

[1170] "Generative AI" is an artificial intelligence technology that generates and renders virtual objects in real time based on user input data and viewpoint information.

[1171] The present invention allows users to customize their own avatars and virtual pets, share them with other users in real time, and use an emotion engine to reflect the user's emotions, providing a more immersive collaborative experience. Specific embodiments are described below.

[1172] overview

[1173] This system allows users to customize avatars and virtual pets using devices such as smartphones or smart glasses, and share them with other users in real time. Furthermore, an emotion engine can recognize users' emotions in real time and reflect them in the avatars and virtual pets.

[1174] Hardware and Software

[1175] Hardware:

[1176] Smartphone, smart glasses, or head-mounted display (e.g. Microsoft HoloLens)

[1177] Cameras and sensors (for obtaining viewpoint information)

[1178] Server (for data storage and processing)

[1179] software:

[1180] OpenCV (image processing library)

[1181] Dlib (Facial Recognition Library)

[1182] EmotionRecognition (emotion recognition engine)

[1183] AvatarCustomization (Avatar Customization Interface)

[1184] ArServerSdk (AR Server SDK)

[1185] Processing flow

[1186] 1. Providing customization methods

[1187] Users use a smartphone or smart glasses interface to customize their avatar and virtual pet, selecting clothing, accessories, and the type and characteristics of their virtual pet.

[1188] 2. Data storage and sharing

[1189] The customized data is sent to the server and stored in a database, and the data is distributed in real time so that other users can share this customized data in the same space.

[1190] 3. Obtaining viewpoint information and drawing

[1191] Cameras and sensors capture the user's viewpoint information and send it to a server. The AI ​​generates instructions for drawing the virtual pet and avatar based on the viewpoint information and sends them to the user's device. The device then draws the images in real time based on these instructions.

[1192] 4. Introducing the Emotion Engine

[1193] The system recognizes emotions from the user's facial expressions and voice and sends them to the server, which uses this information to generate instructions to change the behavior and appearance of the avatar and virtual pet. Emotional information is also shared with other users.

[1194] Specific examples

[1195] For example, imagine a user puts on smart glasses in a brick-and-mortar store and customizes their avatar with a pink dress, a blue hat, and a brown dog. This information is displayed in real time to other users. When the emotion engine recognizes the user's smile, the avatar also changes to a smile, and the change is communicated to other users.

[1196] Prompt Sentence Examples

[1197] "The avatar's outfit is a pink dress, the hat is blue, and the pet is a brown dog. If the user is smiling, the avatar should also smile."

[1198] In this way, the system of the present invention allows users to customize their avatars and virtual pets, sharing emotions with other users in real time, providing an immersive, interactive, and collaborative experience.

[1199] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1200] Step 1:

[1201] The terminal provides a customization interface for users to customize their avatars and virtual pets.

[1202] This includes an interface where the user can select the avatar's clothing, accessories, and the type and characteristics of the virtual pet, and the device generates this as customization data as the user provides the input.

[1203] Input: User selection information (clothing, accessories, pet characteristics)

[1204] Output: Customization data

[1205] Step 2:

[1206] The terminal transmits the generated customization data to the server.

[1207] The server stores the received customization data in a data storage means, so that the customization information is safely recorded and can be reused later.

[1208] Input: Customization data

[1209] Output: Customization data stored in the database

[1210] Step 3:

[1211] The server distributes the saved customization data in real time to other users wearing AR devices in the same space.

[1212] The other users' terminals prepare to render the avatars and virtual pets based on the received customization data.

[1213] Input: Customization data stored in the database

[1214] Output: Customization data distributed to other users' devices

[1215] Step 4:

[1216] The device acquires the user's viewpoint information using a camera or sensor.

[1217] This viewpoint information is sent to the server and used as the basic data for drawing. Based on the user's viewpoint information, the generative AI model creates drawing instructions for the virtual pet and avatar.

[1218] Input: User's viewpoint information (camera image, gaze data)

[1219] Output: Viewpoint information sent to the server

[1220] Step 5:

[1221] The server uses a generative AI model based on the user's viewpoint to generate instructions for rendering the virtual pet and avatar in real time.

[1222] This instruction is sent to the terminal, which displays the avatar and virtual pet in the user's field of view.

[1223] Input: Viewpoint information

[1224] Output: Drawing instructions generated by the generative AI model

[1225] Step 6:

[1226] The device collects data using a camera and microphone to analyze the user's facial expressions and voice.

[1227] The collected data is sent to the emotion engine to recognize the user's emotions, and the recognition results are sent to the server.

[1228] Input: User's facial expression data, voice data

[1229] Output: Recognized emotion data

[1230] Step 7:

[1231] The server generates instructions to change the behavior and appearance of the avatar and the virtual pet based on the emotion information recognized by the emotion engine means.

[1232] These instructions are sent to the device and reflected in the appearance of the avatar and virtual pet in real time.

[1233] Input: Emotion data

[1234] Output: Movement and appearance change instructions

[1235] Step 8:

[1236] The server also shares the recognized emotion information with the terminals of other users.

[1237] Other users can observe in real time how the behavior and appearance of their avatar or virtual pet changes based on the shared emotional information.

[1238] Input: Emotion data

[1239] Output: Emotion data shared with other users' devices

[1240] Specific examples

[1241] This is the process of a user putting on smart glasses in a physical store and customizing their avatar with a pink dress, a blue hat, and a brown dog. The customized information is displayed in real time to other users. When the emotion engine recognizes the user's smile, the avatar also changes to a smile, and the change is communicated to other users.

[1242] An example of a prompt sentence is "The avatar's outfit is a pink dress, the hat is blue, and the pet is a brown dog. When the user smiles, make the avatar smile too."

[1243] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1244] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1245] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1246] [Third embodiment]

[1247] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1248] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1249] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1250] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1251] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1252] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1253] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1254] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1255] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1256] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1257] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1258] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1259] The purpose of the experience sharing AR system of the present invention is to enable multiple users to share the same AR content in real time and interact with each other. An embodiment of this system will be described in detail below.

[1260] overview

[1261] The AR device (terminal) worn by the user has the means to customize the user's avatar and virtual pet and share it with other users, and uses generation AI to render the virtual pet and avatar in real time based on the user's viewpoint, providing the user with an immersive experience.

[1262] Program processing

[1263] 1. Customize your avatar and virtual pet

[1264] The terminal provides the user with an interface for customizing their avatar and virtual pet.

[1265] The user selects the clothes, hats, accessories, type and characteristics of the virtual pet.

[1266] Customized data is generated within the device.

[1267] The terminal transmits the generated customization data to the server.

[1268] 2. Data storage and sharing

[1269] The server stores the received customization data in a database.

[1270] The server distributes the customization data to other users wearing the same AR device.

[1271] The other user's device renders the user's avatar and virtual pet based on the received customization data.

[1272] 3. Real-time drawing

[1273] The device acquires the user's viewpoint information using a camera or sensor.

[1274] The terminal transmits the acquired viewpoint information to the server.

[1275] The server uses a generative AI based on the received viewpoint information to create instructions for drawing a virtual pet or avatar in real time.

[1276] The server sends drawing instructions to the terminal.

[1277] The terminal displays the avatar or virtual pet in the user's field of view based on the received drawing instructions.

[1278] 4. Providing a collaborative experience

[1279] The server distributes information about AR content installed in a specific location (e.g., advertising space on a train).

[1280] The device draws virtual content (e.g., a crane game) at that location based on the received AR content information.

[1281] Users can interact with AR content and share the experience with other users in real time.

[1282] Specific examples

[1283] Example 1: User A and User B are on the same train

[1284] First, user A customizes his or her avatar with a blue shirt and jeans and a virtual pet with a black cat. Once the customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to user B's device. User B's device then renders user A's avatar and the black cat in real time. Similarly, if user B changes his or her avatar to a red hat and sets his or her virtual pet to a white dog, the data is shared using the same procedure.

[1285] Example 2: Virtual "crane game" on a train

[1286] The server sets up a virtual "crane game" in the advertising space on the train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B can participate in the same game and enjoy a collaborative experience by operating the crane in real time.

[1287] In this way, the system of the present invention allows users to share AR content with other users in real time and enjoy an immersive collaborative experience.

[1288] The processing flow will be explained below.

[1289] Below, the program processing of the experience-sharing AR system is explained in concrete steps.

[1290] Customize your avatar and virtual pet

[1291] Step 1:

[1292] The user launches the "Avatar Connect" application.

[1293] The user opens the application and is presented with a login screen.

[1294] Step 2:

[1295] The user enters their login information and logs in.

[1296] The user enters their account information and clicks the "Login" button.

[1297] The application sends the login information to the server.

[1298] Step 3:

[1299] The server authenticates the user's login information.

[1300] The server checks the database to verify the login information.

[1301] The authentication result is returned to the terminal.

[1302] Step 4:

[1303] The terminal displays a customization screen to the user.

[1304] If authentication is successful, the customization interface will be displayed.

[1305] Users can choose the appearance of their avatar and virtual pet.

[1306] Step 5:

[1307] Users customize their avatars and virtual pets.

[1308] Choose from multiple options (clothes, hats, pet types, etc.).

[1309] Confirm your customizations with the "Save" button.

[1310] Step 6:

[1311] The terminal transmits the customization data to the server.

[1312] The terminal generates customization data and transmits it to the server.

[1313] Data storage and sharing

[1314] Step 7:

[1315] The server stores the customization data in a database.

[1316] The received customization data is recorded in a database.

[1317] Step 8:

[1318] The server distributes the data to other users.

[1319] Data is distributed in real time to other users wearing the same AR device.

[1320] Step 9:

[1321] The terminal receives customization information of other users.

[1322] Based on the customization information received by the terminal, the avatars and virtual pets of other users are drawn.

[1323] Real-time drawing

[1324] Step 10:

[1325] The terminal acquires the user's viewpoint information.

[1326] Cameras and sensors are used to obtain the user's viewpoint information (position, direction).

[1327] Step 11:

[1328] The terminal transmits viewpoint information to the server.

[1329] The acquired viewpoint information is sent to the server in real time.

[1330] Step 12:

[1331] The server creates drawing instructions using a generation AI.

[1332] Based on the received viewpoint information, the generation AI generates drawing instructions for the virtual pet or avatar.

[1333] Sends drawing instructions to the device.

[1334] Step 13:

[1335] The terminal performs real-time drawing.

[1336] Based on instructions received from the server, an avatar or virtual pet is displayed in the user's field of view.

[1337] Providing a shared experience

[1338] Step 14:

[1339] The server distributes information about AR content placed in a specific location.

[1340] Virtual content (such as a crane game) is placed in advertising space in a specific location (e.g., inside a train), and that information is sent to all devices.

[1341] Step 15:

[1342] The terminal receives the content information and renders it.

[1343] Based on the received AR content information, virtual content is drawn at that location.

[1344] Step 16:

[1345] Users can share experiences with AR content.

[1346] Multiple users can simultaneously interact with virtual content and enjoy a collaborative experience in real time.

[1347] Example 1

[1348] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1349] In current augmented reality (AR) technology, it has been difficult for users to share their avatars or virtual characters with other users in real time, and even to share experiences collaboratively. Furthermore, there are insufficient means for users to instantly reflect their customized avatars or virtual characters on other users' screens. As a result, the immersive and interactive nature of the AR experience is diminished, resulting in a decrease in user satisfaction.

[1350] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1351] In this invention, the server includes a customization means for allowing a user to customize his or her own avatar and virtual character, a data storage means for storing customization data generated by the customization means, a data sharing means for sharing the customization data stored in the storage means with other users, a viewpoint acquisition means for acquiring user viewpoint information, a drawing means for drawing a virtual character and an avatar using a generating artificial intelligence based on the viewpoint information acquired by the viewpoint acquisition means, a display means for displaying the avatar and virtual character drawn by the drawing means in the user's field of view, and a collaborative experience means for allowing multiple users to share the same augmented reality content in real time and have a collaborative experience. This allows users to experience their own and other users' customized avatars and virtual characters in real time and enjoy a collaborative interactive AR experience.

[1352] 1. An "avatar" is a digital character that represents a user in a virtual environment.

[1353] 2. "Virtual Character" means a digital pet or character that a user can customize and control or display within a virtual environment.

[1354] 3. "Customization Method" means the interface and functionality that allows a User to customize and change the type, characteristics, clothing, accessories, etc. of their Avatar or Virtual Character.

[1355] 4. "Data Storage Means" means a storage device such as a database or memory that stores information about a user's customized avatar and virtual character.

[1356] 5. "Data Sharing Method" means the communication method and functionality for sharing saved customized data with other users.

[1357] 6. "Point of view acquisition means" refers to devices and technologies for acquiring user's point of view information using cameras and sensors.

[1358] 7. “Generative AI” refers to artificial intelligence techniques and models for rendering avatars or virtual characters in real time based on a user’s perspective.

[1359] 8. "Rendering means" means a graphics engine or software that displays an avatar or virtual character in the user's field of view based on the acquired viewpoint information and customization data.

[1360] 9. "Display means" means the technology and devices for displaying a rendered avatar or virtual character on the display of an AR device worn by a user.

[1361] 10. "Collaborative experience means" refers to functions and technologies that enable multiple users to share the same augmented reality content in real time and interact with it collaboratively.

[1362] The present invention provides a system for customizing and sharing avatars and virtual characters in real time with other users using augmented reality (AR) devices worn by users, thereby enabling users to share interactive and immersive experiences with other users. Detailed embodiments of the present invention are described below.

[1363] System Configuration

[1364] This system mainly consists of a customization means, a data storage means, a data sharing means, a viewpoint acquisition means, a drawing means, and a display means. The specific processing of each means is as follows.

[1365] Hardware and software used

[1366] Device: AR device worn by the user (e.g., AR glasses, smartphone)

[1367] Server: Cloud server or on-premise server

[1368] Generative AI model: GAN (generative artificial network)

[1369] Database: MySQL, PostgreSQL

[1370] Communication technology: HTTP, WebSocket

[1371] Graphics API: OpenGL, DirectX

[1372] Rendering engine: Unity, Unreal Engine

[1373] Program processing

[1374] Avatar / Virtual Character Customization

[1375] The device provides the user with an interface for customizing their avatar and virtual character. The user uses this interface to select the avatar's clothing and hat, as well as the type and characteristics of the virtual character. The selected customization data is generated within the device and saved in JSON format or similar.

[1376] Data storage and sharing

[1377] The generated customization data is sent from the device to a server. The server stores the received data in a database and shares it with other users in real time using a data sharing means. The other users' devices analyze the received data and draw the avatar and virtual character.

[1378] Obtaining viewpoint information

[1379] The device acquires the user's viewpoint information using the built-in camera and sensors. This information includes the user's position, orientation, and gaze point. The acquired viewpoint information is sent to the server.

[1380] Real-time drawing

[1381] The server uses a generative AI model based on the received viewpoint information to generate instructions for rendering the virtual character and avatar in real time, which are then sent to the device, which then displays the avatar and virtual character in the user's field of view.

[1382] Providing a shared experience

[1383] The server also distributes information about augmented reality content installed in specific locations, allowing multiple users to share the same content, such as a virtual "crane game" installed on a train. Users can manipulate this content and enjoy a collaborative experience with other users in real time.

[1384] Specific examples

[1385] Example 1: User A and User B are in the same location

[1386] User A customizes his avatar with a blue shirt and jeans and a virtual character with a black cat. Once customization is complete, his device sends this information to the server. The server shares the data with other users and distributes it to User B's device. User B's device then renders User A's avatar and the black cat in real time. Similarly, if User B changes his avatar to a red hat and sets his virtual character to a white dog, the data is shared using the same procedure.

[1387] Example 2: Virtual "crane game" on a train

[1388] The server sets up a virtual "crane game" in the advertising space on the train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B can participate in the same game and enjoy a collaborative experience by operating the crane in real time.

[1389] Prompt Sentence Examples

[1390] "After user A customizes his avatar with a blue shirt and jeans and a virtual character with a black cat, explain how that information is shared with other users in real time via a server."

[1391] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1392] Step 1:

[1393] Avatar / Virtual Character Customization

[1394] The terminal provides the user with an interface for customizing the avatar and virtual character.

[1395] Input: User selection (clothes, hats, accessories, virtual character type and characteristics)

[1396] Output: Custom data (e.g., JSON format)

[1397] The user uses the provided interface to make selections to customize their avatar or virtual character, such as choosing a blue shirt or a black cat.

[1398] This generates customization data within the device. Specifically, the user's selections are constructed as JSON format data.

[1399] Step 2:

[1400] Data storage and sharing

[1401] The terminal transmits the generated customization data to the server.

[1402] Input: Customization data

[1403] Output: Data sent to the server

[1404] The server stores the received customization data in a database.

[1405] Input: Customization data

[1406] Output: Records in the database

[1407] The server distributes the customized data to other users in real time using the data sharing means.

[1408] Input: Saved customization data

[1409] Output: Data distributed to other users

[1410] The other user's device will then render the avatar or virtual character based on the received customization data. Specifically, it will analyze the JSON data and render it in real time using OpenGL, DirectX, etc.

[1411] Step 3:

[1412] Obtaining viewpoint information

[1413] The device acquires the user's viewpoint information using the built-in camera and sensors.

[1414] Input: User's viewpoint movement

[1415] Output: Viewpoint information (position, direction, gaze point)

[1416] The device accurately measures the user's location and gaze direction, and collects this data in real time. Specifically, it combines data from the IMU and GPS to form viewpoint information.

[1417] Step 4:

[1418] Sending viewpoint information

[1419] The terminal transmits the acquired viewpoint information to the server.

[1420] Input: Viewpoint information

[1421] Output: Viewpoint information sent to the server

[1422] The viewpoint information is sent to the server using a high-speed communication protocol (e.g., WebSocket) with minimal latency.

[1423] Step 5:

[1424] Real-time drawing

[1425] The server uses a generation AI based on the received viewpoint information to generate instructions for drawing virtual characters and avatars in real time.

[1426] Input: Viewpoint information, customization data

[1427] Output: Drawing instructions

[1428] Using a generative AI model (e.g., GAN), data for real-time drawing is generated based on the user's viewpoint information and customization data.

[1429] The server transmits the generated drawing instructions to the terminal.

[1430] Input: Drawing instructions

[1431] Output: Drawing instructions sent to the device

[1432] Step 6:

[1433] Displaying avatars and virtual characters

[1434] The terminal displays an avatar or virtual character in the user's field of view based on the received drawing instructions.

[1435] Input: Drawing instructions

[1436] Output: Avatars and virtual characters displayed on the screen

[1437] Use a rendering engine (e.g. Unity, Unreal Engine) to visualize the drawing instructions.

[1438] Step 7:

[1439] Providing a shared experience

[1440] A server distributes information about augmented reality content installed in a specific location.

[1441] Input: Augmented reality content information

[1442] Output: Augmented reality content information delivered to the device

[1443] The terminal renders the virtual content at that location based on the received augmented reality content information.

[1444] Input: Augmented reality content information

[1445] Output: Virtual content displayed on a display

[1446] Users can interact with AR content and share the experience with other users in real time.

[1447] Input: User operation information

[1448] Output: Realizing a collaborative experience

[1449] User operation information is shared with other users via the server, enabling an interactive experience.

[1450] (Application example 1)

[1451] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1452] In modern brick-and-mortar stores, it is difficult for customers to determine which product is most suitable for them when selecting a product. Furthermore, there is a lack of means for real-time interaction when multiple customers are selecting and trying out products together. This results in a poor shopping experience and often leads to suboptimal purchasing decisions. Furthermore, the lack of tools to visually check product details and discount information in real time can discourage purchases.

[1453] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1454] In this invention, the server includes: a customization unit that enables a user to customize their own avatar and virtual pet; a storage unit that stores the customization data generated by the customization unit; a sharing unit that shares the customization data stored in the storage unit with other users; an acquisition unit that acquires user viewpoint information; a drawing unit that uses a generative AI model to draw the virtual pet and avatar based on the viewpoint information acquired by the acquisition unit; a display unit that displays the avatar and virtual pet drawn by the drawing unit in the user's field of view; an information display unit that acquires product information from a physical store and displays it in the user's field of view in real time; and a collaborative experience unit that enables multiple users to try on products together and interact in real time. This allows customers to enjoy a collaborative shopping experience with other customers while visually checking product information and discount information in real time.

[1455] An "avatar" is a digital character or icon that a user uses to represent themselves in a virtual environment.

[1456] A "virtual pet" is a digitally generated animal or character that a user can keep within a virtual environment.

[1457] "Customization means" refers to functions or modules that allow users to change the appearance or attributes of their avatar or virtual pet.

[1458] "Storage" refers to a data storage system for temporarily or permanently storing the generated customization data.

[1459] The "sharing means" is a function or module that transmits the saved customized data to other users and shares it in real time.

[1460] The "acquisition means" is an input device such as a camera or sensor for acquiring user viewpoint information.

[1461] A "generative AI model" is an artificial intelligence algorithm that draws avatars and virtual pets in real time based on the user's viewpoint information and customization data.

[1462] A "rendering tool" is a graphics engine or software that displays an avatar or virtual pet based on data obtained using a generative AI model.

[1463] The "display means" refers to a display device or AR device that displays the rendered avatar or virtual pet in the user's field of view.

[1464] "Information display means" refers to a function or module that acquires product information and discount information from a physical store in real time and displays it in the user's field of view.

[1465] "Collaborative experience means" refers to functions and systems that allow multiple users to try on products simultaneously and interact in real time.

[1466] This invention provides a system that allows users to customize their own avatars and virtual pets and share and interact with other users in real time to improve the customer shopping experience in physical stores. Specific embodiments of this system are described below.

[1467] Hardware and software used

[1468] Smartphones and smart glasses: Capture user perspective information and use it to display customized data.

[1469] Cameras and sensors: Input devices for acquiring user viewpoint information.

[1470] Database Management: Use Amazon RDS (relational data store) to store customized data.

[1471] Viewpoint information processing: We use OpenCV (an open-source computer vision library) to process viewpoint information.

[1472] Generative AI model: Uses Google Cloud AI (generative AI service) to render avatars and virtual pets in real time.

[1473] Real-time communication: Use Firebase (a real-time database) to share data in real time.

[1474] Specific operation of the system

[1475] Customize your avatar and virtual pet

[1476] Users use their smartphones or smart glasses to customize their avatars and virtual pets, including clothing, hats, accessories, and the type and characteristics of their virtual pets. Once customization is complete, the data is stored on a server.

[1477] Data storage and sharing

[1478] The server stores the generated customization data in Amazon RDS, and then shares the data in real time with other users in the same physical store using Firebase.

[1479] Obtaining viewpoint information and drawing in real time

[1480] The cameras and sensors in smartphones and smart glasses capture the user's viewpoint. This information is sent to a server and processed by OpenCV. The server uses a generative AI model to generate real-time instructions for rendering an avatar or virtual pet based on the viewpoint and sends them to each device.

[1481] Displaying product information

[1482] When a user points the camera at a product using a smartphone or smart glasses, the server obtains product information and discount information and displays it in the user's field of view in real time.

[1483] Providing a shared experience

[1484] The server provides functionality for multiple users to collaboratively try on the same product and interact in real time, allowing users to collaboratively select products and make purchasing decisions with other users.

[1485] Specific examples

[1486] Example 1: User A and User B are in the same store

[1487] First, User A customizes his or her avatar with a blue shirt and jeans, and his or her virtual pet with a black cat. Once customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to User B's device. User B's device then renders User A's avatar and the black cat in real time.

[1488] Example 2: Virtual try-on session in a physical store

[1489] The server acquires product information from the physical store and distributes related information to all devices. The devices receive the information and display a virtual try-on opportunity within the user's field of view. User A and User B can virtually try on the same item and check the results in real time, allowing for a collaborative experience.

[1490] Prompt sentence input example

[1491] "Prompt for real-time rendering of avatars and virtual pets from user viewpoint information"

[1492] This will improve the in-store shopping experience and allow customers to make more effective product selections.

[1493] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1494] Step 1:

[1495] A user customizes an avatar and a virtual pet using a smartphone or smart glasses, selecting clothes, hats, accessories, and the type and characteristics of the virtual pet to generate customization data.

[1496] Input: User customization choices (clothes, hats, accessories, virtual pets)

[1497] Output: Customization data (avatar and virtual pet information)

[1498] Step 2:

[1499] The device sends the generated customization data to the server, which stores it in Amazon RDS.

[1500] Input: Customization data

[1501] Output: Saved customization data

[1502] Step 3:

[1503] The server shares the saved customization data in real time with other users in the same physical store, sending the data in real time using Firebase.

[1504] Input: Saved customization data

[1505] Output: Shared customization data

[1506] Step 4:

[1507] The camera and sensors on the device capture the user's viewpoint, which is then sent to the server.

[1508] Input: User's viewpoint information (camera, sensor)

[1509] Output: Viewpoint information sent to the server

[1510] Step 5:

[1511] The server processes the viewpoint information using OpenCV and uses a generative AI model to generate drawing instructions for the avatar and virtual pet based on the viewpoint information.

[1512] Input: Viewpoint information

[1513] Output: Drawing instructions

[1514] Step 6:

[1515] The server generates and sends drawing instructions to each device, which then draws the avatar and virtual pet in real time based on the drawing instructions.

[1516] Input: Drawing instructions

[1517] Output: Real-time rendered avatars and virtual pets

[1518] Step 7:

[1519] The device displays the rendered avatar and virtual pet in the user's field of view, using a display device such as a smartphone or smart glasses.

[1520] Input: Real-time rendered avatars and virtual pets

[1521] Output: Avatar and virtual pet displayed in the user's field of view

[1522] Step 8:

[1523] When a user points the camera at a product in a physical store, the device acquires product information, which is then sent to a server, which then acquires product information and discount information.

[1524] Input: Product information (input from camera)

[1525] Output: Retrieved product information and discount information

[1526] Step 9:

[1527] The server transmits the acquired product information and discount information to the terminal in real time, and the terminal displays the information in the user's field of view.

[1528] Input: Product information and discount information

[1529] Output: Product information and discount information displayed in the user's field of view

[1530] Step 10:

[1531] The server sends information to the devices so that multiple users can try on products simultaneously and interact with each other in real time. Each device displays the try-on information and interaction information in the user's field of view, enabling a collaborative experience.

[1532] Input: Try-on information and interaction information

[1533] Output: Try-on information and interaction information displayed in the user's field of view

[1534] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1535] The purpose of the experience-sharing AR system of the present invention is to provide a more immersive collaborative experience by allowing users to customize their own avatars and virtual pets, share them with other users in real time, and reflect the users' emotions using an emotion engine. An embodiment of this system is described below in detail.

[1536] overview

[1537] The system includes a means for users to customize their own avatars and virtual pets, and a data sharing means for sharing them with other users. It also uses a generation AI to acquire user viewpoint information and, based on that information, render the virtual pets and avatars in real time, displaying them in the user's field of view. It also incorporates an emotion engine that recognizes the user's emotions, and reflects the recognized emotion information in the behavior and appearance of the avatars and virtual pets, providing a more realistic experience.

[1538] Program processing

[1539] 1. Customize your avatar and virtual pet

[1540] The terminal provides the user with an interface for customizing their avatar and virtual pet.

[1541] The user selects the clothes, hats, accessories, type and characteristics of the virtual pet.

[1542] The terminal generates customization data and transmits it to the server.

[1543] 2. Data storage and sharing

[1544] The server stores the received customization data in a database.

[1545] The server distributes the customization data to other users wearing the same AR device.

[1546] The other user's device renders the avatar or virtual pet based on the received customization data.

[1547] 3. Real-time drawing

[1548] The device acquires the user's viewpoint information using a camera or sensor.

[1549] The terminal transmits viewpoint information to the server.

[1550] The server uses a generative AI based on the received viewpoint information to create instructions for drawing a virtual pet or avatar in real time.

[1551] The server sends drawing instructions to the terminal.

[1552] The terminal displays the avatar or virtual pet in the user's field of view based on the received drawing instructions.

[1553] 4. Introducing the Emotion Engine

[1554] The terminal recognizes emotions by analyzing the user's facial expressions, voice, and behavioral patterns.

[1555] The emotion engine sends the recognized emotion information to the server.

[1556] The server generates instructions to change the behavior and appearance of the avatar and virtual pet in real time based on the emotional information.

[1557] The user's emotional information is shared with other users and reflected in the avatars and virtual pets of the other users.

[1558] 5. Providing a collaborative experience

[1559] The server distributes information about AR content installed in a specific location (e.g., advertising space on a train).

[1560] The device draws virtual content (e.g., a crane game) at that location based on the received AR content information.

[1561] Users can interact with AR content and share the experience with other users in real time.

[1562] Specific examples

[1563] Example 1: User A and User B are on the same train

[1564] First, user A customizes his or her avatar with a blue shirt and jeans, and customizes his or her virtual pet with a black cat. Once the customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to user B's device. User B's device then renders user A's avatar and the black cat in real time. Similarly, if user B changes his or her avatar to a red hat and sets his or her virtual pet to a white dog, the data is shared using the same procedure. Furthermore, the emotion engine recognizes user A's smile and reflects it by making the avatar smile.

[1565] Example 2: Virtual "crane game" on a train

[1566] The server sets up a virtual "crane game" in an advertising space on a train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B participate in the crane game and operate the crane in real time. The emotion engine recognizes User B's excitement, and the avatar's behavior changes to match that emotion.

[1567] In this way, the system of the present invention allows users to share AR content with other users in real time and enjoy a immersive collaborative experience that reflects their emotions.

[1568] The processing flow will be explained below.

[1569] Below, we will explain the processing of the program of the invention, which combines an experience-sharing AR system with an emotion engine, by dividing it into specific steps.

[1570] Customize your avatar and virtual pet

[1571] Step 1:

[1572] The user launches the "Avatar Connect" application.

[1573] The user opens the application and is presented with a login screen.

[1574] Step 2:

[1575] The user enters their login information and logs in.

[1576] The user enters their account information and clicks the "Login" button.

[1577] The application sends the login information to the server.

[1578] Step 3:

[1579] The server authenticates the user's login information.

[1580] The server checks the database to verify the login information.

[1581] The authentication result is returned to the terminal.

[1582] Step 4:

[1583] The terminal displays a customization screen to the user.

[1584] If authentication is successful, the customization interface will be displayed.

[1585] Users can choose the appearance of their avatar and virtual pet.

[1586] Step 5:

[1587] Users customize their avatars and virtual pets.

[1588] Choose from multiple options (clothes, hats, pet types, etc.).

[1589] Confirm your customizations with the "Save" button.

[1590] Step 6:

[1591] The terminal transmits the customization data to the server.

[1592] The terminal generates customization data and transmits it to the server.

[1593] Data storage and sharing

[1594] Step 7:

[1595] The server stores the customization data in a database.

[1596] The received customization data is recorded in a database.

[1597] Step 8:

[1598] The server distributes the data to other users.

[1599] Data is distributed in real time to other users wearing the same AR device.

[1600] Step 9:

[1601] The terminal receives customization information of other users.

[1602] Based on the customization information received by the terminal, the avatars and virtual pets of other users are drawn.

[1603] Real-time drawing

[1604] Step 10:

[1605] The terminal acquires the user's viewpoint information.

[1606] Cameras and sensors are used to obtain the user's viewpoint information (position, direction).

[1607] Step 11:

[1608] The terminal transmits viewpoint information to the server.

[1609] The acquired viewpoint information is sent to the server in real time.

[1610] Step 12:

[1611] The server creates drawing instructions using a generation AI.

[1612] Based on the received viewpoint information, the generation AI generates drawing instructions for the virtual pet or avatar.

[1613] Sends drawing instructions to the device.

[1614] Step 13:

[1615] The terminal performs real-time drawing.

[1616] Based on instructions received from the server, an avatar or virtual pet is displayed in the user's field of view.

[1617] Introducing the Emotion Engine

[1618] Step 14:

[1619] The device analyzes the user's facial expressions, voice, and behavioral patterns.

[1620] The built-in camera, microphone, and motion sensor are used to obtain user emotional information.

[1621] Step 15:

[1622] The emotion engine analyzes the recognized emotion information.

[1623] The emotion engine analyzes the acquired information and identifies the user's emotional state.

[1624] Step 16:

[1625] The emotion engine sends the emotion information to the server.

[1626] Emotional information is sent to the server in real time.

[1627] Step 17:

[1628] The server generates instructions to change the behavior and appearance of the avatar and virtual pet based on the emotional information.

[1629] Based on the user's emotional information, instructions are generated to change the behavior and appearance of the avatar or virtual pet in real time.

[1630] Step 18:

[1631] The server shares the emotion information with other users.

[1632] Emotional information is also distributed to other users who share the experience in the same space.

[1633] Step 19:

[1634] The device receives the emotional information and changes the behavior and appearance of the avatar and virtual pet.

[1635] The behavior and appearance of the avatar and virtual pet are changed based on the received emotional information.

[1636] Providing a shared experience

[1637] Step 20:

[1638] The server distributes information about AR content placed in a specific location.

[1639] Virtual content (such as a crane game) is placed in a specific location (e.g., advertising space on a train), and that information is sent to all devices.

[1640] Step 21:

[1641] The terminal receives the content information and renders it.

[1642] Based on the received AR content information, virtual content is drawn at that location.

[1643] Step 22:

[1644] Users can share experiences with AR content.

[1645] Multiple users can simultaneously interact with virtual content and enjoy a collaborative experience in real time.

[1646] Example 2

[1647] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1648] While current AR systems allow users to customize their avatars and virtual pets, they struggle to share them in real time and provide a more immersive experience by incorporating emotions. They also lack a means to effectively use generative AI models to render images in real time based on the user's viewpoint.

[1649] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: customization means for allowing a user to customize his or her own avatar and virtual pet; data storage means for storing customization data generated by the customization means; data sharing means for sharing the customization data stored in the storage means with other users; viewpoint acquisition means for acquiring user viewpoint information; drawing means for drawing the virtual pet and avatar using a generative AI model based on the viewpoint information acquired by the viewpoint acquisition means; display means for displaying the avatar and virtual pet drawn by the drawing means in the user's field of view; emotion recognition means for analyzing the user's facial expressions, voice, and behavior patterns to recognize emotions; and emotion reflection means for changing the behavior and appearance of the avatar and virtual pet in real time based on the user's emotion recognized by the emotion recognition means. This allows users to share their avatars and virtual pets with other users in real time, providing a more realistic experience with emotions reflected.

[1650] "Customization means" refers to a means by which a user can freely change and set their own avatar and virtual pet.

[1651] The "data storage means" is a means for storing the customization data generated by the customization means.

[1652] The "data sharing means" is a means for sharing the customized data stored in the storage means with other users.

[1653] The "viewpoint acquisition means" is a means for acquiring user viewpoint information.

[1654] A "generative AI model" is an artificial intelligence model that renders virtual pets and avatars in real time based on the user's viewpoint information.

[1655] The "drawing means" is a means for drawing a virtual pet and an avatar using a generating AI model based on the viewpoint information acquired by the viewpoint acquisition means.

[1656] The "display means" is a means for displaying the avatar and virtual pet drawn by the drawing means in the user's field of view.

[1657] The "emotion recognition means" is a means for recognizing emotions by analyzing the user's facial expressions, voice, and behavioral patterns.

[1658] The "emotion reflection means" is a means for changing the behavior and appearance of the avatar and virtual pet in real time based on the user's emotion recognized by the emotion recognition means.

[1659] MODE FOR CARRYING OUT THE INVENTION

[1660] The present invention is an experience-sharing AR system that allows users to customize avatars and virtual pets in real time, share them with other users, and reflect emotions to provide an immersive experience. The system includes the following main means:

[1661] Customization methods

[1662] The device provides the user with an interface for customizing their avatar and virtual pet. The user selects clothing, accessories, and the type of virtual pet, and customization data is generated based on this selection. This data can then be shared with other users via a data sharing mechanism.

[1663] Data storage means

[1664] The server stores the customization data received from the terminal in a database, which centrally manages the customization information for each user.

[1665] Data sharing methods

[1666] The server distributes the stored customization data in real time to other users wearing augmented reality devices in the same space. The other users' devices then render avatars and virtual pets based on the received customization data.

[1667] Perspective acquisition method

[1668] The device captures the user's viewpoint using a camera or sensor, which is then sent to a server where it is used for real-time rendering using a generative AI model.

[1669] Generative AI model and rendering method

[1670] The server uses the generative AI model based on the viewpoint information acquired by the viewpoint acquisition means to create instructions for drawing the virtual pet and avatar, which then causes the virtual pet and avatar to be displayed in the user's field of view in real time.

[1671] Display means

[1672] Based on the drawing instructions received by the device, the avatar or virtual pet is displayed within the user's field of view, allowing the user to see their customized character in the AR space.

[1673] Emotion recognition and reflection measures

[1674] The device analyzes the user's facial expressions, voice, and behavioral patterns to recognize emotions. The emotion information obtained by the emotion recognition means is sent to a server, which generates instructions to change the behavior and appearance of the avatar and virtual pet in real time based on that information. This allows the user's emotions to be reflected in the avatar and virtual pet, providing a more realistic experience.

[1675] Specific examples

[1676] Example 1: User A and User B are on the same train

[1677] First, user A customizes his or her avatar with a blue shirt and jeans, and customizes his or her virtual pet with a black cat. Once customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to user B's device. User B's device then uses the data to draw the avatar and black cat in real time. The emotion engine recognizes user A's smile and reflects it so that the avatar smiles.

[1678] Example 2: Virtual "crane game" on a train

[1679] The server sets up a virtual "crane game" in an advertising space on a train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B participate in the crane game and operate the crane in real time. The emotion engine recognizes User B's excitement, and the avatar's behavior changes to match that emotion.

[1680] Prompt Sentence Examples

[1681] "How would the system behave if a user changed their avatar to a red shirt and blue jeans and their virtual pet to a white dog?"

[1682] "Please explain in detail the steps to install a virtual crane game inside a car."

[1683] In this way, the system of the present invention allows users to customize, share, and even emote avatars and virtual pets in real time, providing an immersive AR experience.

[1684] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1685] Program processing flow and specific explanation

[1686] Step 1: Display the avatar / virtual pet customization interface

[1687] The terminal provides the user with an interface for customizing their avatar and virtual pet.

[1688] Input: Application launch and user selection actions.

[1689] Output: The interface is displayed on the device display.

[1690] What it does: A screen appears on your smartphone or tablet where you can select clothes, accessories, and a virtual pet for your avatar.

[1691] Step 2: User selects options

[1692] The user operates the interface to select clothes, hats, accessories, and the type and characteristics of the virtual pet.

[1693] Input: User taps and swipes.

[1694] Output: Data for the selected option.

[1695] What happens: The user taps to select an option, such as a blue shirt, jeans, or a black cat.

[1696] Step 3: Generate customization data

[1697] The terminal generates customization data based on the selected options.

[1698] Input: Option data selected by the user.

[1699] Output: Customization data (e.g. JSON file).

[1700] Specific behavior: Information about the selected options is compiled into a single JSON file.

[1701] Step 4: Submit customization data

[1702] The terminal transmits the customization data to the server.

[1703] Input: Customization data.

[1704] Output: Sending data to the server.

[1705] What happens: A JSON file is uploaded to a server over the internet.

[1706] Step 5: Save your data

[1707] The server stores the received customization data in a database.

[1708] Input: The submitted customization data.

[1709] Output: Customization data stored in a database.

[1710] What happens: The contents of the JSON file are inserted into a specific table in the database.

[1711] Step 6: Serving Data

[1712] The server distributes the stored customization data in real time to other users wearing augmented reality devices in the same space.

[1713] Input: Customization data stored in the database.

[1714] Output: Data sent to other users' devices.

[1715] Specific operation: Customized data extracted from the database is pushed to other users' devices.

[1716] Step 7: Obtaining viewpoint information

[1717] The device acquires the user's viewpoint information using a camera or sensor.

[1718] Input: Camera video data and sensor location information.

[1719] Output: Viewpoint information.

[1720] Specific operations: The camera captures images, and sensors detect position and tilt.

[1721] Step 8: Sending viewpoint information

[1722] The terminal transmits viewpoint information to the server.

[1723] Input: Obtained viewpoint information.

[1724] Output: Send viewpoint information to the server.

[1725] Specific operation: Location information and image data are uploaded to the server.

[1726] Step 9: Generate drawing instructions

[1727] The server uses a generative AI model based on the viewpoint information to create instructions for drawing the virtual pet and avatar.

[1728] Input: viewpoint information.

[1729] Output: Drawing instruction data.

[1730] Specific operation: The generative AI analyzes viewpoint information and generates new drawing guidance.

[1731] Step 10: Sending drawing instructions

[1732] The server sends drawing instructions to the terminal.

[1733] Input: Drawing instruction data.

[1734] Output: Sending drawing instructions to the device.

[1735] Specific behavior: The newly generated drawing instruction data is pushed to the device.

[1736] Step 11: Drawing within the field of view

[1737] Based on the drawing instructions received by the terminal, an avatar or virtual pet is displayed in the user's field of view.

[1738] Input: Drawing instruction data.

[1739] Output: An avatar or virtual pet displayed in the user's field of view.

[1740] What happens: The device display updates with the new drawing.

[1741] Step 12: Emotion Recognition

[1742] The device analyzes the user's facial expressions, voice, and behavioral patterns to recognize emotions.

[1743] Input: Camera footage, audio data, behavioral data.

[1744] Output: Recognized emotion information.

[1745] How it works: The camera captures facial expressions, the microphone records audio, and this information is analyzed by analytics software.

[1746] Step 13: Sending Emotional Information

[1747] The emotion engine sends the recognized emotion information to the server.

[1748] Input: Emotion information.

[1749] Output: Sending emotion information to the server.

[1750] Specific operation: Emotion information data is uploaded to the server.

[1751] Step 14: Instructions for behavior and appearance changes

[1752] The server generates instructions for changing the behavior and appearance of the avatar and virtual pet in real time based on the emotional information.

[1753] Input: Emotion information.

[1754] Output: Instruction data for changing behavior and appearance.

[1755] Specific behavior: The server generates instructions for behavior and appearance changes based on the new emotion.

[1756] Step 15: Sharing Emotional Information

[1757] The server shares the user's emotional information with other users and reflects it in the avatars and virtual pets of the other users.

[1758] Input: Emotion information.

[1759] Output: Sending emotion information to other users' devices.

[1760] Specific operation: Emotion information is pushed to other users' devices.

[1761] Step 16: Drawing after reflecting emotions

[1762] The device updates the behavior and appearance of the avatar or virtual pet based on the emotional information and redraws it in the user's field of view.

[1763] Input: Data instructing behavior and appearance changes.

[1764] Output: Updated view of your avatar and virtual pet.

[1765] What happens: The device display will update to show the new emotion-based drawing.

[1766] This process allows users to share their customized avatars and virtual pets with other users in real time, creating a more emotionally immersive experience.

[1767] (Application example 2)

[1768] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1769] In today's physical store experience, it is difficult for customers to share emotions or customized characters with other customers in real time, and individual experiences tend to be emphasized. This results in a lack of interaction and collaborative experiences between customers in the store, and a lack of realism and enjoyment. Furthermore, a shopping experience using dynamic avatars or virtual pets that reflect emotions has not yet been realized.

[1770] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1771] In this invention, the server includes customization means for allowing users to customize their own avatars and virtual pets, data storage means for storing customization data generated by the customization means, data sharing means for sharing the customization data stored in the storage means with other users, viewpoint acquisition means for acquiring user viewpoint information, drawing means for using a generation AI to draw the virtual pet and avatar based on the viewpoint information acquired by the viewpoint acquisition means, display means for displaying the avatar and virtual pet drawn by the drawing means in the user's field of view, emotion engine means for detecting the user's emotions and changing the behavior and appearance of the avatar and virtual pet based on the emotions, and emotion sharing means for sharing the emotion information detected by the emotion engine means with other users. This allows users to share their emotions with other customers in real time, providing a dynamic and interactive shopping experience.

[1772] A "customization means" is a method that provides an interface for a user to individually configure their avatar and virtual pet, and select and edit their appearance and behavior.

[1773] "Data storage means" refers to a database or storage system for temporarily or permanently recording user-customized avatar and virtual pet settings.

[1774] The "data sharing means" is a communication means for transmitting the customization data generated by the customization means to other users in real time or with a certain time lag.

[1775] The "viewpoint acquisition means" is a device including a camera or sensor for collecting information about the direction and field of view of the user.

[1776] "Rendering means" refers to a method for displaying virtual pets and avatars superimposed on the real world using a generating AI based on the user's viewpoint information.

[1777] The "display means" is a technology for displaying the avatar and virtual pet generated by the drawing means on the display of the user's device (such as a smartphone or smart glasses).

[1778] The "emotion engine means" is a system that analyzes the user's facial expressions, voice, and behavioral patterns to recognize emotions, and reflects this emotional information in the avatar or virtual pet.

[1779] The "emotion sharing means" is a communication means for transmitting and sharing the user's emotion information recognized by the emotion engine means with other users.

[1780] "Generative AI" is an artificial intelligence technology that generates and renders virtual objects in real time based on user input data and viewpoint information.

[1781] The present invention allows users to customize their own avatars and virtual pets, share them with other users in real time, and use an emotion engine to reflect the user's emotions, providing a more immersive collaborative experience. Specific embodiments are described below.

[1782] overview

[1783] This system allows users to customize avatars and virtual pets using devices such as smartphones or smart glasses, and share them with other users in real time. Furthermore, an emotion engine can recognize users' emotions in real time and reflect them in the avatars and virtual pets.

[1784] Hardware and Software

[1785] Hardware:

[1786] Smartphone, smart glasses, or head-mounted display (e.g. Microsoft HoloLens)

[1787] Cameras and sensors (for obtaining viewpoint information)

[1788] Server (for data storage and processing)

[1789] software:

[1790] OpenCV (image processing library)

[1791] Dlib (Facial Recognition Library)

[1792] EmotionRecognition (emotion recognition engine)

[1793] AvatarCustomization (Avatar Customization Interface)

[1794] ArServerSdk (AR Server SDK)

[1795] Processing flow

[1796] 1. Providing customization methods

[1797] Users use a smartphone or smart glasses interface to customize their avatar and virtual pet, selecting clothing, accessories, and the type and characteristics of their virtual pet.

[1798] 2. Data storage and sharing

[1799] The customized data is sent to the server and stored in a database, and the data is distributed in real time so that other users can share this customized data in the same space.

[1800] 3. Obtaining viewpoint information and drawing

[1801] Cameras and sensors capture the user's viewpoint information and send it to a server. The AI ​​generates instructions for drawing the virtual pet and avatar based on the viewpoint information and sends them to the user's device. The device then draws the images in real time based on these instructions.

[1802] 4. Introducing the Emotion Engine

[1803] The system recognizes emotions from the user's facial expressions and voice and sends them to the server, which uses this information to generate instructions to change the behavior and appearance of the avatar and virtual pet. Emotional information is also shared with other users.

[1804] Specific examples

[1805] For example, imagine a user puts on smart glasses in a brick-and-mortar store and customizes their avatar with a pink dress, a blue hat, and a brown dog. This information is displayed in real time to other users. When the emotion engine recognizes the user's smile, the avatar also changes to a smile, and the change is communicated to other users.

[1806] Prompt Sentence Examples

[1807] "The avatar's outfit is a pink dress, the hat is blue, and the pet is a brown dog. If the user is smiling, the avatar should also smile."

[1808] In this way, the system of the present invention allows users to customize their avatars and virtual pets, sharing emotions with other users in real time, providing an immersive, interactive, and collaborative experience.

[1809] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1810] Step 1:

[1811] The terminal provides a customization interface for users to customize their avatars and virtual pets.

[1812] This includes an interface where the user can select the avatar's clothing, accessories, and the type and characteristics of the virtual pet, and the device generates this as customization data as the user provides the input.

[1813] Input: User selection information (clothing, accessories, pet characteristics)

[1814] Output: Customization data

[1815] Step 2:

[1816] The terminal transmits the generated customization data to the server.

[1817] The server stores the received customization data in a data storage means, so that the customization information is safely recorded and can be reused later.

[1818] Input: Customization data

[1819] Output: Customization data stored in the database

[1820] Step 3:

[1821] The server distributes the saved customization data in real time to other users wearing AR devices in the same space.

[1822] The other users' terminals prepare to render the avatars and virtual pets based on the received customization data.

[1823] Input: Customization data stored in the database

[1824] Output: Customization data distributed to other users' devices

[1825] Step 4:

[1826] The device acquires the user's viewpoint information using a camera or sensor.

[1827] This viewpoint information is sent to the server and used as the basic data for drawing. Based on the user's viewpoint information, the generative AI model creates drawing instructions for the virtual pet and avatar.

[1828] Input: User's viewpoint information (camera image, gaze data)

[1829] Output: Viewpoint information sent to the server

[1830] Step 5:

[1831] The server uses a generative AI model based on the user's viewpoint to generate instructions for rendering the virtual pet and avatar in real time.

[1832] This instruction is sent to the terminal, which displays the avatar and virtual pet in the user's field of view.

[1833] Input: Viewpoint information

[1834] Output: Drawing instructions generated by the generative AI model

[1835] Step 6:

[1836] The device collects data using a camera and microphone to analyze the user's facial expressions and voice.

[1837] The collected data is sent to the emotion engine to recognize the user's emotions, and the recognition results are sent to the server.

[1838] Input: User's facial expression data, voice data

[1839] Output: Recognized emotion data

[1840] Step 7:

[1841] The server generates instructions to change the behavior and appearance of the avatar and the virtual pet based on the emotion information recognized by the emotion engine means.

[1842] These instructions are sent to the device and reflected in the appearance of the avatar and virtual pet in real time.

[1843] Input: Emotion data

[1844] Output: Movement and appearance change instructions

[1845] Step 8:

[1846] The server also shares the recognized emotion information with the terminals of other users.

[1847] Other users can observe in real time how the behavior and appearance of their avatar or virtual pet changes based on the shared emotional information.

[1848] Input: Emotion data

[1849] Output: Emotion data shared with other users' devices

[1850] Specific examples

[1851] This is the process of a user putting on smart glasses in a physical store and customizing their avatar with a pink dress, a blue hat, and a brown dog. The customized information is displayed in real time to other users. When the emotion engine recognizes the user's smile, the avatar also changes to a smile, and the change is communicated to other users.

[1852] An example of a prompt sentence is "The avatar's outfit is a pink dress, the hat is blue, and the pet is a brown dog. When the user smiles, make the avatar smile too."

[1853] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1854] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1855] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1856] [Fourth embodiment]

[1857] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1858] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1859] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1860] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1861] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1862] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1863] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1864] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1865] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1866] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1867] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1868] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1869] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1870] The purpose of the experience sharing AR system of the present invention is to enable multiple users to share the same AR content in real time and interact with each other. An embodiment of this system will be described in detail below.

[1871] overview

[1872] The AR device (terminal) worn by the user has the means to customize the user's avatar and virtual pet and share it with other users, and uses generation AI to render the virtual pet and avatar in real time based on the user's viewpoint, providing the user with an immersive experience.

[1873] Program processing

[1874] 1. Customize your avatar and virtual pet

[1875] The terminal provides the user with an interface for customizing their avatar and virtual pet.

[1876] The user selects the clothes, hats, accessories, type and characteristics of the virtual pet.

[1877] Customized data is generated within the device.

[1878] The terminal transmits the generated customization data to the server.

[1879] 2. Data storage and sharing

[1880] The server stores the received customization data in a database.

[1881] The server distributes the customization data to other users wearing the same AR device.

[1882] The other user's device renders the user's avatar and virtual pet based on the received customization data.

[1883] 3. Real-time drawing

[1884] The device acquires the user's viewpoint information using a camera or sensor.

[1885] The terminal transmits the acquired viewpoint information to the server.

[1886] The server uses a generative AI based on the received viewpoint information to create instructions for drawing a virtual pet or avatar in real time.

[1887] The server sends drawing instructions to the terminal.

[1888] The terminal displays the avatar or virtual pet in the user's field of view based on the received drawing instructions.

[1889] 4. Providing a collaborative experience

[1890] The server distributes information about AR content installed in a specific location (e.g., advertising space on a train).

[1891] The device draws virtual content (e.g., a crane game) at that location based on the received AR content information.

[1892] Users can interact with AR content and share the experience with other users in real time.

[1893] Specific examples

[1894] Example 1: User A and User B are on the same train

[1895] First, user A customizes his or her avatar with a blue shirt and jeans and a virtual pet with a black cat. Once the customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to user B's device. User B's device then renders user A's avatar and the black cat in real time. Similarly, if user B changes his or her avatar to a red hat and sets his or her virtual pet to a white dog, the data is shared using the same procedure.

[1896] Example 2: Virtual "crane game" on a train

[1897] The server sets up a virtual "crane game" in the advertising space on the train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B can participate in the same game and enjoy a collaborative experience by operating the crane in real time.

[1898] In this way, the system of the present invention allows users to share AR content with other users in real time and enjoy an immersive collaborative experience.

[1899] The processing flow will be explained below.

[1900] Below, the program processing of the experience-sharing AR system is explained in concrete steps.

[1901] Customize your avatar and virtual pet

[1902] Step 1:

[1903] The user launches the "Avatar Connect" application.

[1904] The user opens the application and is presented with a login screen.

[1905] Step 2:

[1906] The user enters their login information and logs in.

[1907] The user enters their account information and clicks the "Login" button.

[1908] The application sends the login information to the server.

[1909] Step 3:

[1910] The server authenticates the user's login information.

[1911] The server checks the database to verify the login information.

[1912] The authentication result is returned to the terminal.

[1913] Step 4:

[1914] The terminal displays a customization screen to the user.

[1915] If authentication is successful, the customization interface will be displayed.

[1916] Users can choose the appearance of their avatar and virtual pet.

[1917] Step 5:

[1918] Users customize their avatars and virtual pets.

[1919] Choose from multiple options (clothes, hats, pet types, etc.).

[1920] Confirm your customizations with the "Save" button.

[1921] Step 6:

[1922] The terminal transmits the customization data to the server.

[1923] The terminal generates customization data and transmits it to the server.

[1924] Data storage and sharing

[1925] Step 7:

[1926] The server stores the customization data in a database.

[1927] The received customization data is recorded in a database.

[1928] Step 8:

[1929] The server distributes the data to other users.

[1930] Data is distributed in real time to other users wearing the same AR device.

[1931] Step 9:

[1932] The terminal receives customization information of other users.

[1933] Based on the customization information received by the terminal, the avatars and virtual pets of other users are drawn.

[1934] Real-time drawing

[1935] Step 10:

[1936] The terminal acquires the user's viewpoint information.

[1937] Cameras and sensors are used to obtain the user's viewpoint information (position, direction).

[1938] Step 11:

[1939] The terminal transmits viewpoint information to the server.

[1940] The acquired viewpoint information is sent to the server in real time.

[1941] Step 12:

[1942] The server creates drawing instructions using a generation AI.

[1943] Based on the received viewpoint information, the generation AI generates drawing instructions for the virtual pet or avatar.

[1944] Sends drawing instructions to the device.

[1945] Step 13:

[1946] The terminal performs real-time drawing.

[1947] Based on instructions received from the server, an avatar or virtual pet is displayed in the user's field of view.

[1948] Providing a shared experience

[1949] Step 14:

[1950] The server distributes information about AR content placed in a specific location.

[1951] Virtual content (such as a crane game) is placed in advertising space in a specific location (e.g., inside a train), and that information is sent to all devices.

[1952] Step 15:

[1953] The terminal receives the content information and renders it.

[1954] Based on the received AR content information, virtual content is drawn at that location.

[1955] Step 16:

[1956] Users can share experiences with AR content.

[1957] Multiple users can simultaneously interact with virtual content and enjoy a collaborative experience in real time.

[1958] Example 1

[1959] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1960] In current augmented reality (AR) technology, it has been difficult for users to share their avatars or virtual characters with other users in real time, and even to share experiences collaboratively. Furthermore, there are insufficient means for users to instantly reflect their customized avatars or virtual characters on other users' screens. As a result, the immersive and interactive nature of the AR experience is diminished, resulting in a decrease in user satisfaction.

[1961] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1962] In this invention, the server includes a customization means for allowing a user to customize his or her own avatar and virtual character, a data storage means for storing customization data generated by the customization means, a data sharing means for sharing the customization data stored in the storage means with other users, a viewpoint acquisition means for acquiring user viewpoint information, a drawing means for drawing a virtual character and an avatar using a generating artificial intelligence based on the viewpoint information acquired by the viewpoint acquisition means, a display means for displaying the avatar and virtual character drawn by the drawing means in the user's field of view, and a collaborative experience means for allowing multiple users to share the same augmented reality content in real time and have a collaborative experience. This allows users to experience their own and other users' customized avatars and virtual characters in real time and enjoy a collaborative interactive AR experience.

[1963] 1. An "avatar" is a digital character that represents a user in a virtual environment.

[1964] 2. "Virtual Character" means a digital pet or character that a user can customize and control or display within a virtual environment.

[1965] 3. "Customization Method" means the interface and functionality that allows a User to customize and change the type, characteristics, clothing, accessories, etc. of their Avatar or Virtual Character.

[1966] 4. "Data Storage Means" means a storage device such as a database or memory that stores information about a user's customized avatar and virtual character.

[1967] 5. "Data Sharing Method" means the communication method and functionality for sharing saved customized data with other users.

[1968] 6. "Point of view acquisition means" refers to devices and technologies for acquiring user's point of view information using cameras and sensors.

[1969] 7. “Generative AI” refers to artificial intelligence techniques and models for rendering avatars or virtual characters in real time based on a user’s perspective.

[1970] 8. "Rendering means" means a graphics engine or software that displays an avatar or virtual character in the user's field of view based on the acquired viewpoint information and customization data.

[1971] 9. "Display means" means the technology and devices for displaying a rendered avatar or virtual character on the display of an AR device worn by a user.

[1972] 10. "Collaborative experience means" refers to functions and technologies that enable multiple users to share the same augmented reality content in real time and interact with it collaboratively.

[1973] The present invention provides a system for customizing and sharing avatars and virtual characters in real time with other users using augmented reality (AR) devices worn by users, thereby enabling users to share interactive and immersive experiences with other users. Detailed embodiments of the present invention are described below.

[1974] System Configuration

[1975] This system mainly consists of a customization means, a data storage means, a data sharing means, a viewpoint acquisition means, a drawing means, and a display means. The specific processing of each means is as follows.

[1976] Hardware and software used

[1977] Device: AR device worn by the user (e.g., AR glasses, smartphone)

[1978] Server: Cloud server or on-premise server

[1979] Generative AI model: GAN (generative artificial network)

[1980] Database: MySQL, PostgreSQL

[1981] Communication technology: HTTP, WebSocket

[1982] Graphics API: OpenGL, DirectX

[1983] Rendering engine: Unity, Unreal Engine

[1984] Program processing

[1985] Avatar / Virtual Character Customization

[1986] The device provides the user with an interface for customizing their avatar and virtual character. The user uses this interface to select the avatar's clothing and hat, as well as the type and characteristics of the virtual character. The selected customization data is generated within the device and saved in JSON format or similar.

[1987] Data storage and sharing

[1988] The generated customization data is sent from the device to a server. The server stores the received data in a database and shares it with other users in real time using a data sharing means. The other users' devices analyze the received data and draw the avatar and virtual character.

[1989] Obtaining viewpoint information

[1990] The device acquires the user's viewpoint information using the built-in camera and sensors. This information includes the user's position, orientation, and gaze point. The acquired viewpoint information is sent to the server.

[1991] Real-time drawing

[1992] The server uses a generative AI model based on the received viewpoint information to generate instructions for rendering the virtual character and avatar in real time, which are then sent to the device, which then displays the avatar and virtual character in the user's field of view.

[1993] Providing a shared experience

[1994] The server also distributes information about augmented reality content installed in specific locations, allowing multiple users to share the same content, such as a virtual "crane game" installed on a train. Users can manipulate this content and enjoy a collaborative experience with other users in real time.

[1995] Specific examples

[1996] Example 1: User A and User B are in the same location

[1997] User A customizes his avatar with a blue shirt and jeans and a virtual character with a black cat. Once customization is complete, his device sends this information to the server. The server shares the data with other users and distributes it to User B's device. User B's device then renders User A's avatar and the black cat in real time. Similarly, if User B changes his avatar to a red hat and sets his virtual character to a white dog, the data is shared using the same procedure.

[1998] Example 2: Virtual "crane game" on a train

[1999] The server sets up a virtual "crane game" in the advertising space on the train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B can participate in the same game and enjoy a collaborative experience by operating the crane in real time.

[2000] Prompt Sentence Examples

[2001] "After user A customizes his avatar with a blue shirt and jeans and a virtual character with a black cat, explain how that information is shared with other users in real time via a server."

[2002] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2003] Step 1:

[2004] Avatar / Virtual Character Customization

[2005] The terminal provides the user with an interface for customizing the avatar and virtual character.

[2006] Input: User selection (clothes, hats, accessories, virtual character type and characteristics)

[2007] Output: Custom data (e.g., JSON format)

[2008] The user uses the provided interface to make selections to customize their avatar or virtual character, such as choosing a blue shirt or a black cat.

[2009] This generates customization data within the device. Specifically, the user's selections are constructed as JSON format data.

[2010] Step 2:

[2011] Data storage and sharing

[2012] The terminal transmits the generated customization data to the server.

[2013] Input: Customization data

[2014] Output: Data sent to the server

[2015] The server stores the received customization data in a database.

[2016] Input: Customization data

[2017] Output: Records in the database

[2018] The server distributes the customized data to other users in real time using the data sharing means.

[2019] Input: Saved customization data

[2020] Output: Data distributed to other users

[2021] The other user's device will then render the avatar or virtual character based on the received customization data. Specifically, it will analyze the JSON data and render it in real time using OpenGL, DirectX, etc.

[2022] Step 3:

[2023] Obtaining viewpoint information

[2024] The device acquires the user's viewpoint information using the built-in camera and sensors.

[2025] Input: User's viewpoint movement

[2026] Output: Viewpoint information (position, direction, gaze point)

[2027] The device accurately measures the user's location and gaze direction, and collects this data in real time. Specifically, it combines data from the IMU and GPS to form viewpoint information.

[2028] Step 4:

[2029] Sending viewpoint information

[2030] The terminal transmits the acquired viewpoint information to the server.

[2031] Input: Viewpoint information

[2032] Output: Viewpoint information sent to the server

[2033] The viewpoint information is sent to the server using a high-speed communication protocol (e.g., WebSocket) with minimal latency.

[2034] Step 5:

[2035] Real-time drawing

[2036] The server uses a generation AI based on the received viewpoint information to generate instructions for drawing virtual characters and avatars in real time.

[2037] Input: Viewpoint information, customization data

[2038] Output: Drawing instructions

[2039] Using a generative AI model (e.g., GAN), data for real-time drawing is generated based on the user's viewpoint information and customization data.

[2040] The server transmits the generated drawing instructions to the terminal.

[2041] Input: Drawing instructions

[2042] Output: Drawing instructions sent to the device

[2043] Step 6:

[2044] Displaying avatars and virtual characters

[2045] The terminal displays an avatar or virtual character in the user's field of view based on the received drawing instructions.

[2046] Input: Drawing instructions

[2047] Output: Avatars and virtual characters displayed on the screen

[2048] Use a rendering engine (e.g. Unity, Unreal Engine) to visualize the drawing instructions.

[2049] Step 7:

[2050] Providing a shared experience

[2051] A server distributes information about augmented reality content installed in a specific location.

[2052] Input: Augmented reality content information

[2053] Output: Augmented reality content information delivered to the device

[2054] The terminal renders the virtual content at that location based on the received augmented reality content information.

[2055] Input: Augmented reality content information

[2056] Output: Virtual content displayed on a display

[2057] Users can interact with AR content and share the experience with other users in real time.

[2058] Input: User operation information

[2059] Output: Realizing a collaborative experience

[2060] User operation information is shared with other users via the server, enabling an interactive experience.

[2061] (Application example 1)

[2062] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2063] In modern brick-and-mortar stores, it is difficult for customers to determine which product is most suitable for them when selecting a product. Furthermore, there is a lack of means for real-time interaction when multiple customers are selecting and trying out products together. This results in a poor shopping experience and often leads to suboptimal purchasing decisions. Furthermore, the lack of tools to visually check product details and discount information in real time can discourage purchases.

[2064] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2065] In this invention, the server includes: a customization unit that enables a user to customize their own avatar and virtual pet; a storage unit that stores the customization data generated by the customization unit; a sharing unit that shares the customization data stored in the storage unit with other users; an acquisition unit that acquires user viewpoint information; a drawing unit that uses a generative AI model to draw the virtual pet and avatar based on the viewpoint information acquired by the acquisition unit; a display unit that displays the avatar and virtual pet drawn by the drawing unit in the user's field of view; an information display unit that acquires product information from a physical store and displays it in the user's field of view in real time; and a collaborative experience unit that enables multiple users to try on products together and interact in real time. This allows customers to enjoy a collaborative shopping experience with other customers while visually checking product information and discount information in real time.

[2066] An "avatar" is a digital character or icon that a user uses to represent themselves in a virtual environment.

[2067] A "virtual pet" is a digitally generated animal or character that a user can keep within a virtual environment.

[2068] "Customization means" refers to functions or modules that allow users to change the appearance or attributes of their avatar or virtual pet.

[2069] "Storage" refers to a data storage system for temporarily or permanently storing the generated customization data.

[2070] The "sharing means" is a function or module that transmits the saved customized data to other users and shares it in real time.

[2071] The "acquisition means" is an input device such as a camera or sensor for acquiring user viewpoint information.

[2072] A "generative AI model" is an artificial intelligence algorithm that draws avatars and virtual pets in real time based on the user's viewpoint information and customization data.

[2073] A "rendering tool" is a graphics engine or software that displays an avatar or virtual pet based on data obtained using a generative AI model.

[2074] The "display means" refers to a display device or AR device that displays the rendered avatar or virtual pet in the user's field of view.

[2075] "Information display means" refers to a function or module that acquires product information and discount information from a physical store in real time and displays it in the user's field of view.

[2076] "Collaborative experience means" refers to functions and systems that allow multiple users to try on products simultaneously and interact in real time.

[2077] This invention provides a system that allows users to customize their own avatars and virtual pets and share and interact with other users in real time to improve the customer shopping experience in physical stores. Specific embodiments of this system are described below.

[2078] Hardware and software used

[2079] Smartphones and smart glasses: Capture user perspective information and use it to display customized data.

[2080] Cameras and sensors: Input devices for acquiring user viewpoint information.

[2081] Database Management: Use Amazon RDS (relational data store) to store customized data.

[2082] Viewpoint information processing: We use OpenCV (an open-source computer vision library) to process viewpoint information.

[2083] Generative AI model: Uses Google Cloud AI (generative AI service) to render avatars and virtual pets in real time.

[2084] Real-time communication: Use Firebase (a real-time database) to share data in real time.

[2085] Specific operation of the system

[2086] Customize your avatar and virtual pet

[2087] Users use their smartphones or smart glasses to customize their avatars and virtual pets, including clothing, hats, accessories, and the type and characteristics of their virtual pets. Once customization is complete, the data is stored on a server.

[2088] Data storage and sharing

[2089] The server stores the generated customization data in Amazon RDS, and then shares the data in real time with other users in the same physical store using Firebase.

[2090] Obtaining viewpoint information and drawing in real time

[2091] The cameras and sensors in smartphones and smart glasses capture the user's viewpoint. This information is sent to a server and processed by OpenCV. The server uses a generative AI model to generate real-time instructions for rendering an avatar or virtual pet based on the viewpoint and sends them to each device.

[2092] Displaying product information

[2093] When a user points the camera at a product using a smartphone or smart glasses, the server obtains product information and discount information and displays it in the user's field of view in real time.

[2094] Providing a shared experience

[2095] The server provides functionality for multiple users to collaboratively try on the same product and interact in real time, allowing users to collaboratively select products and make purchasing decisions with other users.

[2096] Specific examples

[2097] Example 1: User A and User B are in the same store

[2098] First, User A customizes his or her avatar with a blue shirt and jeans, and his or her virtual pet with a black cat. Once customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to User B's device. User B's device then renders User A's avatar and the black cat in real time.

[2099] Example 2: Virtual try-on session in a physical store

[2100] The server acquires product information from the physical store and distributes related information to all devices. The devices receive the information and display a virtual try-on opportunity within the user's field of view. User A and User B can virtually try on the same item and check the results in real time, allowing for a collaborative experience.

[2101] Prompt sentence input example

[2102] "Prompt for real-time rendering of avatars and virtual pets from user viewpoint information"

[2103] This will improve the in-store shopping experience and allow customers to make more effective product selections.

[2104] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2105] Step 1:

[2106] A user customizes an avatar and a virtual pet using a smartphone or smart glasses, selecting clothes, hats, accessories, and the type and characteristics of the virtual pet to generate customization data.

[2107] Input: User customization choices (clothes, hats, accessories, virtual pets)

[2108] Output: Customization data (avatar and virtual pet information)

[2109] Step 2:

[2110] The device sends the generated customization data to the server, which stores it in Amazon RDS.

[2111] Input: Customization data

[2112] Output: Saved customization data

[2113] Step 3:

[2114] The server shares the saved customization data in real time with other users in the same physical store, sending the data in real time using Firebase.

[2115] Input: Saved customization data

[2116] Output: Shared customization data

[2117] Step 4:

[2118] The camera and sensors on the device capture the user's viewpoint, which is then sent to the server.

[2119] Input: User's viewpoint information (camera, sensor)

[2120] Output: Viewpoint information sent to the server

[2121] Step 5:

[2122] The server processes the viewpoint information using OpenCV and uses a generative AI model to generate drawing instructions for the avatar and virtual pet based on the viewpoint information.

[2123] Input: Viewpoint information

[2124] Output: Drawing instructions

[2125] Step 6:

[2126] The server generates and sends drawing instructions to each device, which then draws the avatar and virtual pet in real time based on the drawing instructions.

[2127] Input: Drawing instructions

[2128] Output: Real-time rendered avatars and virtual pets

[2129] Step 7:

[2130] The device displays the rendered avatar and virtual pet in the user's field of view, using a display device such as a smartphone or smart glasses.

[2131] Input: Real-time rendered avatars and virtual pets

[2132] Output: Avatar and virtual pet displayed in the user's field of view

[2133] Step 8:

[2134] When a user points the camera at a product in a physical store, the device acquires product information, which is then sent to a server, which then acquires product information and discount information.

[2135] Input: Product information (input from camera)

[2136] Output: Retrieved product information and discount information

[2137] Step 9:

[2138] The server transmits the acquired product information and discount information to the terminal in real time, and the terminal displays the information in the user's field of view.

[2139] Input: Product information and discount information

[2140] Output: Product information and discount information displayed in the user's field of view

[2141] Step 10:

[2142] The server sends information to the devices so that multiple users can try on products simultaneously and interact with each other in real time. Each device displays the try-on information and interaction information in the user's field of view, enabling a collaborative experience.

[2143] Input: Try-on information and interaction information

[2144] Output: Try-on information and interaction information displayed in the user's field of view

[2145] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2146] The purpose of the experience-sharing AR system of the present invention is to provide a more immersive collaborative experience by allowing users to customize their own avatars and virtual pets, share them with other users in real time, and reflect the users' emotions using an emotion engine. An embodiment of this system is described below in detail.

[2147] overview

[2148] The system includes a means for users to customize their own avatars and virtual pets, and a data sharing means for sharing them with other users. It also uses a generation AI to acquire user viewpoint information and, based on that information, render the virtual pets and avatars in real time, displaying them in the user's field of view. It also incorporates an emotion engine that recognizes the user's emotions, and reflects the recognized emotion information in the behavior and appearance of the avatars and virtual pets, providing a more realistic experience.

[2149] Program processing

[2150] 1. Customize your avatar and virtual pet

[2151] The terminal provides the user with an interface for customizing their avatar and virtual pet.

[2152] The user selects the clothes, hats, accessories, type and characteristics of the virtual pet.

[2153] The terminal generates customization data and transmits it to the server.

[2154] 2. Data storage and sharing

[2155] The server stores the received customization data in a database.

[2156] The server distributes the customization data to other users wearing the same AR device.

[2157] The other user's device renders the avatar or virtual pet based on the received customization data.

[2158] 3. Real-time drawing

[2159] The device acquires the user's viewpoint information using a camera or sensor.

[2160] The terminal transmits viewpoint information to the server.

[2161] The server uses a generative AI based on the received viewpoint information to create instructions for drawing a virtual pet or avatar in real time.

[2162] The server sends drawing instructions to the terminal.

[2163] The terminal displays the avatar or virtual pet in the user's field of view based on the received drawing instructions.

[2164] 4. Introducing the Emotion Engine

[2165] The terminal recognizes emotions by analyzing the user's facial expressions, voice, and behavioral patterns.

[2166] The emotion engine sends the recognized emotion information to the server.

[2167] The server generates instructions to change the behavior and appearance of the avatar and virtual pet in real time based on the emotional information.

[2168] The user's emotional information is shared with other users and reflected in the avatars and virtual pets of the other users.

[2169] 5. Providing a collaborative experience

[2170] The server distributes information about AR content installed in a specific location (e.g., advertising space on a train).

[2171] The device draws virtual content (e.g., a crane game) at that location based on the received AR content information.

[2172] Users can interact with AR content and share the experience with other users in real time.

[2173] Specific examples

[2174] Example 1: User A and User B are on the same train

[2175] First, user A customizes his or her avatar with a blue shirt and jeans, and customizes his or her virtual pet with a black cat. Once the customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to user B's device. User B's device then renders user A's avatar and the black cat in real time. Similarly, if user B changes his or her avatar to a red hat and sets his or her virtual pet to a white dog, the data is shared using the same procedure. Furthermore, the emotion engine recognizes user A's smile and reflects it by making the avatar smile.

[2176] Example 2: Virtual "crane game" on a train

[2177] The server sets up a virtual "crane game" in an advertising space on a train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B participate in the crane game and operate the crane in real time. The emotion engine recognizes User B's excitement, and the avatar's behavior changes to match that emotion.

[2178] In this way, the system of the present invention allows users to share AR content with other users in real time and enjoy a immersive collaborative experience that reflects their emotions.

[2179] The processing flow will be explained below.

[2180] Below, we will explain the processing of the program of the invention, which combines an experience-sharing AR system with an emotion engine, by dividing it into specific steps.

[2181] Customize your avatar and virtual pet

[2182] Step 1:

[2183] The user launches the "Avatar Connect" application.

[2184] The user opens the application and is presented with a login screen.

[2185] Step 2:

[2186] The user enters their login information and logs in.

[2187] The user enters their account information and clicks the "Login" button.

[2188] The application sends the login information to the server.

[2189] Step 3:

[2190] The server authenticates the user's login information.

[2191] The server checks the database to verify the login information.

[2192] The authentication result is returned to the terminal.

[2193] Step 4:

[2194] The terminal displays a customization screen to the user.

[2195] If authentication is successful, the customization interface will be displayed.

[2196] Users can choose the appearance of their avatar and virtual pet.

[2197] Step 5:

[2198] Users customize their avatars and virtual pets.

[2199] Choose from multiple options (clothes, hats, pet types, etc.).

[2200] Confirm your customizations with the "Save" button.

[2201] Step 6:

[2202] The terminal transmits the customization data to the server.

[2203] The terminal generates customization data and transmits it to the server.

[2204] Data storage and sharing

[2205] Step 7:

[2206] The server stores the customization data in a database.

[2207] The received customization data is recorded in a database.

[2208] Step 8:

[2209] The server distributes the data to other users.

[2210] Data is distributed in real time to other users wearing the same AR device.

[2211] Step 9:

[2212] The terminal receives customization information of other users.

[2213] Based on the customization information received by the terminal, the avatars and virtual pets of other users are drawn.

[2214] Real-time drawing

[2215] Step 10:

[2216] The terminal acquires the user's viewpoint information.

[2217] Cameras and sensors are used to obtain the user's viewpoint information (position, direction).

[2218] Step 11:

[2219] The terminal transmits viewpoint information to the server.

[2220] The acquired viewpoint information is sent to the server in real time.

[2221] Step 12:

[2222] The server creates drawing instructions using a generation AI.

[2223] Based on the received viewpoint information, the generation AI generates drawing instructions for the virtual pet or avatar.

[2224] Sends drawing instructions to the device.

[2225] Step 13:

[2226] The terminal performs real-time drawing.

[2227] Based on instructions received from the server, an avatar or virtual pet is displayed in the user's field of view.

[2228] Introducing the Emotion Engine

[2229] Step 14:

[2230] The device analyzes the user's facial expressions, voice, and behavioral patterns.

[2231] The built-in camera, microphone, and motion sensor are used to obtain user emotional information.

[2232] Step 15:

[2233] The emotion engine analyzes the recognized emotion information.

[2234] The emotion engine analyzes the acquired information and identifies the user's emotional state.

[2235] Step 16:

[2236] The emotion engine sends the emotion information to the server.

[2237] Emotional information is sent to the server in real time.

[2238] Step 17:

[2239] The server generates instructions to change the behavior and appearance of the avatar and virtual pet based on the emotional information.

[2240] Based on the user's emotional information, instructions are generated to change the behavior and appearance of the avatar or virtual pet in real time.

[2241] Step 18:

[2242] The server shares the emotion information with other users.

[2243] Emotional information is also distributed to other users who share the experience in the same space.

[2244] Step 19:

[2245] The device receives the emotional information and changes the behavior and appearance of the avatar and virtual pet.

[2246] The behavior and appearance of the avatar and virtual pet are changed based on the received emotional information.

[2247] Providing a shared experience

[2248] Step 20:

[2249] The server distributes information about AR content placed in a specific location.

[2250] Virtual content (such as a crane game) is placed in a specific location (e.g., advertising space on a train), and that information is sent to all devices.

[2251] Step 21:

[2252] The terminal receives the content information and renders it.

[2253] Based on the received AR content information, virtual content is drawn at that location.

[2254] Step 22:

[2255] Users can share experiences with AR content.

[2256] Multiple users can simultaneously interact with virtual content and enjoy a collaborative experience in real time.

[2257] Example 2

[2258] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2259] While current AR systems allow users to customize their avatars and virtual pets, they struggle to share them in real time and provide a more immersive experience by incorporating emotions. They also lack a means to effectively use generative AI models to render images in real time based on the user's viewpoint.

[2260] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: customization means for allowing a user to customize his or her own avatar and virtual pet; data storage means for storing customization data generated by the customization means; data sharing means for sharing the customization data stored in the storage means with other users; viewpoint acquisition means for acquiring user viewpoint information; drawing means for drawing the virtual pet and avatar using a generative AI model based on the viewpoint information acquired by the viewpoint acquisition means; display means for displaying the avatar and virtual pet drawn by the drawing means in the user's field of view; emotion recognition means for analyzing the user's facial expressions, voice, and behavior patterns to recognize emotions; and emotion reflection means for changing the behavior and appearance of the avatar and virtual pet in real time based on the user's emotion recognized by the emotion recognition means. This allows users to share their avatars and virtual pets with other users in real time, providing a more realistic experience with emotions reflected.

[2261] "Customization means" refers to a means by which a user can freely change and set their own avatar and virtual pet.

[2262] The "data storage means" is a means for storing the customization data generated by the customization means.

[2263] The "data sharing means" is a means for sharing the customized data stored in the storage means with other users.

[2264] The "viewpoint acquisition means" is a means for acquiring user viewpoint information.

[2265] A "generative AI model" is an artificial intelligence model that renders virtual pets and avatars in real time based on the user's viewpoint information.

[2266] The "drawing means" is a means for drawing a virtual pet and an avatar using a generating AI model based on the viewpoint information acquired by the viewpoint acquisition means.

[2267] The "display means" is a means for displaying the avatar and virtual pet drawn by the drawing means in the user's field of view.

[2268] The "emotion recognition means" is a means for recognizing emotions by analyzing the user's facial expressions, voice, and behavioral patterns.

[2269] The "emotion reflection means" is a means for changing the behavior and appearance of the avatar and virtual pet in real time based on the user's emotion recognized by the emotion recognition means.

[2270] MODE FOR CARRYING OUT THE INVENTION

[2271] The present invention is an experience-sharing AR system that allows users to customize avatars and virtual pets in real time, share them with other users, and reflect emotions to provide an immersive experience. The system includes the following main means:

[2272] Customization methods

[2273] The device provides the user with an interface for customizing their avatar and virtual pet. The user selects clothing, accessories, and the type of virtual pet, and customization data is generated based on this selection. This data can then be shared with other users via a data sharing mechanism.

[2274] Data storage means

[2275] The server stores the customization data received from the terminal in a database, which centrally manages the customization information for each user.

[2276] Data sharing methods

[2277] The server distributes the stored customization data in real time to other users wearing augmented reality devices in the same space. The other users' devices then render avatars and virtual pets based on the received customization data.

[2278] Perspective acquisition method

[2279] The device captures the user's viewpoint using a camera or sensor, which is then sent to a server where it is used for real-time rendering using a generative AI model.

[2280] Generative AI model and rendering method

[2281] The server uses the generative AI model based on the viewpoint information acquired by the viewpoint acquisition means to create instructions for drawing the virtual pet and avatar, which then causes the virtual pet and avatar to be displayed in the user's field of view in real time.

[2282] Display means

[2283] Based on the drawing instructions received by the device, the avatar or virtual pet is displayed within the user's field of view, allowing the user to see their customized character in the AR space.

[2284] Emotion recognition and reflection measures

[2285] The device analyzes the user's facial expressions, voice, and behavioral patterns to recognize emotions. The emotion information obtained by the emotion recognition means is sent to a server, which generates instructions to change the behavior and appearance of the avatar and virtual pet in real time based on that information. This allows the user's emotions to be reflected in the avatar and virtual pet, providing a more realistic experience.

[2286] Specific examples

[2287] Example 1: User A and User B are on the same train

[2288] First, user A customizes his or her avatar with a blue shirt and jeans, and customizes his or her virtual pet with a black cat. Once customization is complete, the device sends this information to the server. The server shares the data with other users and distributes it to user B's device. User B's device then uses the data to draw the avatar and black cat in real time. The emotion engine recognizes user A's smile and reflects it so that the avatar smiles.

[2289] Example 2: Virtual "crane game" on a train

[2290] The server sets up a virtual "crane game" in an advertising space on a train and distributes related information to all devices. The devices receive the information and render the crane game within the user's field of view. User A and User B participate in the crane game and operate the crane in real time. The emotion engine recognizes User B's excitement, and the avatar's behavior changes to match that emotion.

[2291] Prompt Sentence Examples

[2292] "How would the system behave if a user changed their avatar to a red shirt and blue jeans and their virtual pet to a white dog?"

[2293] "Please explain in detail the steps to install a virtual crane game inside a car."

[2294] In this way, the system of the present invention allows users to customize, share, and even emote avatars and virtual pets in real time, providing an immersive AR experience.

[2295] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2296] Program processing flow and specific explanation

[2297] Step 1: Display the avatar / virtual pet customization interface

[2298] The terminal provides the user with an interface for customizing their avatar and virtual pet.

[2299] Input: Application launch and user selection actions.

[2300] Output: The interface is displayed on the device display.

[2301] What it does: A screen appears on your smartphone or tablet where you can select clothes, accessories, and a virtual pet for your avatar.

[2302] Step 2: User selects options

[2303] The user operates the interface to select clothes, hats, accessories, and the type and characteristics of the virtual pet.

[2304] Input: User taps and swipes.

[2305] Output: Data for the selected option.

[2306] What happens: The user taps to select an option, such as a blue shirt, jeans, or a black cat.

[2307] Step 3: Generate customization data

[2308] The terminal generates customization data based on the selected options.

[2309] Input: Option data selected by the user.

[2310] Output: Customization data (e.g. JSON file).

[2311] Specific behavior: Information about the selected options is compiled into a single JSON file.

[2312] Step 4: Submit customization data

[2313] The terminal transmits the customization data to the server.

[2314] Input: Customization data.

[2315] Output: Sending data to the server.

[2316] What happens: A JSON file is uploaded to a server over the internet.

[2317] Step 5: Save your data

[2318] The server stores the received customization data in a database.

[2319] Input: The submitted customization data.

[2320] Output: Customization data stored in a database.

[2321] What happens: The contents of the JSON file are inserted into a specific table in the database.

[2322] Step 6: Serving Data

[2323] The server distributes the stored customization data in real time to other users wearing augmented reality devices in the same space.

[2324] Input: Customization data stored in the database.

[2325] Output: Data sent to other users' devices.

[2326] Specific operation: Customized data extracted from the database is pushed to other users' devices.

[2327] Step 7: Obtaining viewpoint information

[2328] The device acquires the user's viewpoint information using a camera or sensor.

[2329] Input: Camera video data and sensor location information.

[2330] Output: Viewpoint information.

[2331] Specific operations: The camera captures images, and sensors detect position and tilt.

[2332] Step 8: Sending viewpoint information

[2333] The terminal transmits viewpoint information to the server.

[2334] Input: Obtained viewpoint information.

[2335] Output: Send viewpoint information to the server.

[2336] Specific operation: Location information and image data are uploaded to the server.

[2337] Step 9: Generate drawing instructions

[2338] The server uses a generative AI model based on the viewpoint information to create instructions for drawing the virtual pet and avatar.

[2339] Input: viewpoint information.

[2340] Output: Drawing instruction data.

[2341] Specific operation: The generative AI analyzes viewpoint information and generates new drawing guidance.

[2342] Step 10: Sending drawing instructions

[2343] The server sends drawing instructions to the terminal.

[2344] Input: Drawing instruction data.

[2345] Output: Sending drawing instructions to the device.

[2346] Specific behavior: The newly generated drawing instruction data is pushed to the device.

[2347] Step 11: Drawing within the field of view

[2348] Based on the drawing instructions received by the terminal, an avatar or virtual pet is displayed in the user's field of view.

[2349] Input: Drawing instruction data.

[2350] Output: An avatar or virtual pet displayed in the user's field of view.

[2351] What happens: The device display updates with the new drawing.

[2352] Step 12: Emotion Recognition

[2353] The device analyzes the user's facial expressions, voice, and behavioral patterns to recognize emotions.

[2354] Input: Camera footage, audio data, behavioral data.

[2355] Output: Recognized emotion information.

[2356] How it works: The camera captures facial expressions, the microphone records audio, and this information is analyzed by analytics software.

[2357] Step 13: Sending Emotional Information

[2358] The emotion engine sends the recognized emotion information to the server.

[2359] Input: Emotion information.

[2360] Output: Sending emotion information to the server.

[2361] Specific operation: Emotion information data is uploaded to the server.

[2362] Step 14: Instructions for behavior and appearance changes

[2363] The server generates instructions for changing the behavior and appearance of the avatar and virtual pet in real time based on the emotional information.

[2364] Input: Emotion information.

[2365] Output: Instruction data for changing behavior and appearance.

[2366] Specific behavior: The server generates instructions for behavior and appearance changes based on the new emotion.

[2367] Step 15: Sharing Emotional Information

[2368] The server shares the user's emotional information with other users and reflects it in the avatars and virtual pets of the other users.

[2369] Input: Emotion information.

[2370] Output: Sending emotion information to other users' devices.

[2371] Specific operation: Emotion information is pushed to other users' devices.

[2372] Step 16: Drawing after reflecting emotions

[2373] The device updates the behavior and appearance of the avatar or virtual pet based on the emotional information and redraws it in the user's field of view.

[2374] Input: Data instructing behavior and appearance changes.

[2375] Output: Updated view of your avatar and virtual pet.

[2376] What happens: The device display will update to show the new emotion-based drawing.

[2377] This process allows users to share their customized avatars and virtual pets with other users in real time, creating a more emotionally immersive experience.

[2378] (Application example 2)

[2379] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2380] In today's physical store experience, it is difficult for customers to share emotions or customized characters with other customers in real time, and individual experiences tend to be emphasized. This results in a lack of interaction and collaborative experiences between customers in the store, and a lack of realism and enjoyment. Furthermore, a shopping experience using dynamic avatars or virtual pets that reflect emotions has not yet been realized.

[2381] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2382] In this invention, the server includes customization means for allowing users to customize their own avatars and virtual pets, data storage means for storing customization data generated by the customization means, data sharing means for sharing the customization data stored in the storage means with other users, viewpoint acquisition means for acquiring user viewpoint information, drawing means for using a generation AI to draw the virtual pet and avatar based on the viewpoint information acquired by the viewpoint acquisition means, display means for displaying the avatar and virtual pet drawn by the drawing means in the user's field of view, emotion engine means for detecting the user's emotions and changing the behavior and appearance of the avatar and virtual pet based on the emotions, and emotion sharing means for sharing the emotion information detected by the emotion engine means with other users. This allows users to share their emotions with other customers in real time, providing a dynamic and interactive shopping experience.

[2383] A "customization means" is a method that provides an interface for a user to individually configure their avatar and virtual pet, and select and edit their appearance and behavior.

[2384] "Data storage means" refers to a database or storage system for temporarily or permanently recording user-customized avatar and virtual pet settings.

[2385] The "data sharing means" is a communication means for transmitting the customization data generated by the customization means to other users in real time or with a certain time lag.

[2386] The "viewpoint acquisition means" is a device including a camera or sensor for collecting information about the direction and field of view of the user.

[2387] "Rendering means" refers to a method for displaying virtual pets and avatars superimposed on the real world using a generating AI based on the user's viewpoint information.

[2388] The "display means" is a technology for displaying the avatar and virtual pet generated by the drawing means on the display of the user's device (such as a smartphone or smart glasses).

[2389] The "emotion engine means" is a system that analyzes the user's facial expressions, voice, and behavioral patterns to recognize emotions, and reflects this emotional information in the avatar or virtual pet.

[2390] The "emotion sharing means" is a communication means for transmitting and sharing the user's emotion information recognized by the emotion engine means with other users.

[2391] "Generative AI" is an artificial intelligence technology that generates and renders virtual objects in real time based on user input data and viewpoint information.

[2392] The present invention allows users to customize their own avatars and virtual pets, share them with other users in real time, and use an emotion engine to reflect the user's emotions, providing a more immersive collaborative experience. Specific embodiments are described below.

[2393] overview

[2394] This system allows users to customize avatars and virtual pets using devices such as smartphones or smart glasses, and share them with other users in real time. Furthermore, an emotion engine can recognize users' emotions in real time and reflect them in the avatars and virtual pets.

[2395] Hardware and Software

[2396] Hardware:

[2397] Smartphone, smart glasses, or head-mounted display (e.g. Microsoft HoloLens)

[2398] Cameras and sensors (for obtaining viewpoint information)

[2399] Server (for data storage and processing)

[2400] software:

[2401] OpenCV (image processing library)

[2402] Dlib (Facial Recognition Library)

[2403] EmotionRecognition (emotion recognition engine)

[2404] AvatarCustomization (Avatar Customization Interface)

[2405] ArServerSdk (AR Server SDK)

[2406] Processing flow

[2407] 1. Providing customization methods

[2408] Users use a smartphone or smart glasses interface to customize their avatar and virtual pet, selecting clothing, accessories, and the type and characteristics of their virtual pet.

[2409] 2. Data storage and sharing

[2410] The customized data is sent to the server and stored in a database, and the data is distributed in real time so that other users can share this customized data in the same space.

[2411] 3. Obtaining viewpoint information and drawing

[2412] Cameras and sensors capture the user's viewpoint information and send it to a server. The AI ​​generates instructions for drawing the virtual pet and avatar based on the viewpoint information and sends them to the user's device. The device then draws the images in real time based on these instructions.

[2413] 4. Introducing the Emotion Engine

[2414] The system recognizes emotions from the user's facial expressions and voice and sends them to the server, which uses this information to generate instructions to change the behavior and appearance of the avatar and virtual pet. Emotional information is also shared with other users.

[2415] Specific examples

[2416] For example, imagine a user puts on smart glasses in a brick-and-mortar store and customizes their avatar with a pink dress, a blue hat, and a brown dog. This information is displayed in real time to other users. When the emotion engine recognizes the user's smile, the avatar also changes to a smile, and the change is communicated to other users.

[2417] Prompt Sentence Examples

[2418] "The avatar's outfit is a pink dress, the hat is blue, and the pet is a brown dog. If the user is smiling, the avatar should also smile."

[2419] In this way, the system of the present invention allows users to customize their avatars and virtual pets, sharing emotions with other users in real time, providing an immersive, interactive, and collaborative experience.

[2420] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2421] Step 1:

[2422] The terminal provides a customization interface for users to customize their avatars and virtual pets.

[2423] This includes an interface where the user can select the avatar's clothing, accessories, and the type and characteristics of the virtual pet, and the device generates this as customization data as the user provides the input.

[2424] Input: User selection information (clothing, accessories, pet characteristics)

[2425] Output: Customization data

[2426] Step 2:

[2427] The terminal transmits the generated customization data to the server.

[2428] The server stores the received customization data in a data storage means, so that the customization information is safely recorded and can be reused later.

[2429] Input: Customization data

[2430] Output: Customization data stored in the database

[2431] Step 3:

[2432] The server distributes the saved customization data in real time to other users wearing AR devices in the same space.

[2433] The other users' terminals prepare to render the avatars and virtual pets based on the received customization data.

[2434] Input: Customization data stored in the database

[2435] Output: Customization data distributed to other users' devices

[2436] Step 4:

[2437] The device acquires the user's viewpoint information using a camera or sensor.

[2438] This viewpoint information is sent to the server and used as the basic data for drawing. Based on the user's viewpoint information, the generative AI model creates drawing instructions for the virtual pet and avatar.

[2439] Input: User's viewpoint information (camera image, gaze data)

[2440] Output: Viewpoint information sent to the server

[2441] Step 5:

[2442] The server uses a generative AI model based on the user's viewpoint to generate instructions for rendering the virtual pet and avatar in real time.

[2443] This instruction is sent to the terminal, which displays the avatar and virtual pet in the user's field of view.

[2444] Input: Viewpoint information

[2445] Output: Drawing instructions generated by the generative AI model

[2446] Step 6:

[2447] The device collects data using a camera and microphone to analyze the user's facial expressions and voice.

[2448] The collected data is sent to the emotion engine to recognize the user's emotions, and the recognition results are sent to the server.

[2449] Input: User's facial expression data, voice data

[2450] Output: Recognized emotion data

[2451] Step 7:

[2452] The server generates instructions to change the behavior and appearance of the avatar and the virtual pet based on the emotion information recognized by the emotion engine means.

[2453] These instructions are sent to the device and reflected in the appearance of the avatar and virtual pet in real time.

[2454] Input: Emotion data

[2455] Output: Movement and appearance change instructions

[2456] Step 8:

[2457] The server also shares the recognized emotion information with the terminals of other users.

[2458] Other users can observe in real time how the behavior and appearance of their avatar or virtual pet changes based on the shared emotional information.

[2459] Input: Emotion data

[2460] Output: Emotion data shared with other users' devices

[2461] Specific examples

[2462] This is the process of a user putting on smart glasses in a physical store and customizing their avatar with a pink dress, a blue hat, and a brown dog. The customized information is displayed in real time to other users. When the emotion engine recognizes the user's smile, the avatar also changes to a smile, and the change is communicated to other users.

[2463] An example of a prompt sentence is "The avatar's outfit is a pink dress, the hat is blue, and the pet is a brown dog. When the user smiles, make the avatar smile too."

[2464] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2465] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2466] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2467] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2468] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotion...

Claims

1. customization means for allowing a user to customize their avatar and virtual pet; a data storage means for storing the customization data generated by the customization means; a data sharing means for sharing the customized data stored in the storage means with other users; viewpoint acquisition means for acquiring viewpoint information of a user; a drawing means for drawing a virtual pet and an avatar using a generation AI based on the viewpoint information acquired by the viewpoint acquisition means; a display means for displaying the avatar and the virtual pet drawn by the drawing means in the user's field of view; A system including:

2. The system according to claim 1 , wherein the data sharing means distributes the customized data in real time to other users wearing AR devices in the same space.

3. The system according to claim 1 , wherein the drawing means is for sharing information about AR content installed in a specific location and allowing multiple users to have a joint experience in real time.

Images