system

Abstract

We provide the system. [Solution] A means for generating character data converted into a digital format, A means for generating content for virtual reality or augmented reality based on character information converted into a digital format, A means of delivering the generated content to the corresponding platform, A system that includes means for verifying user access rights to requested content and recording access logs.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present invention is to provide a system that effectively and efficiently generates and distributes content when converting existing characters into a digital format and using them as virtual reality (VR) or augmented reality (AR) content. In conventional systems, a great deal of labor and cost are required for conversion into a digital format, content generation, and user management, and it is difficult to exclusively manage content. Therefore, there has been a demand for a system that efficiently and intensively performs these processes.

Means for Solving the Problems

[0005] To solve the above problems, the present invention provides a system that includes the following means. First, it provides means for collecting character data from content providers and converting it into a digital format. Next, it includes means for generating virtual reality (VR) or augmented reality (AR) content based on the character information converted into a digital format. Then, it provides means for distributing the generated content to a dedicated application or online platform. Furthermore, by including means for verifying access rights to content requested by users and recording access, the system streamlines overall management and enables exclusive distribution of content.

[0006] A "digital format" is a data format that converts analog data into a format that can be handled by a computer.

[0007] "Character data" refers to information about characters that appear in media such as manga and anime.

[0008] "Virtual reality (VR)" refers to a technology or experience that allows users to immerse themselves in environments and situations different from the physical world, generated by computer technology.

[0009] Augmented reality (AR) is a technology that overlays digital information onto the real physical environment.

[0010] A "content provider" is a company or organization that provides characters, media, and information.

[0011] A "dedicated application" is software designed specifically for a particular purpose or function.

[0012] An "online platform" is the foundation for services and applications delivered via the internet.

[0013] "Access rights" refer to the rights a user has to use specific digital content.

[0014] An "access log" is a record of when a user accesses a system.

[0015] A "system" refers to an integrated mechanism or setup in which multiple elements or functions work together in coordination. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0020] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] The system for carrying out the present invention will now be described. This system consists of the following main components.

[0038] 1. Character data collection and digitization

[0039] The server first receives character data from content providers (e.g., manga and anime publishers, game companies, etc.). This includes information such as character images, models, voices, and movements. The received character data is then converted into a digital format. This conversion process uses technologies such as scanning to digitize hand-drawn illustrations or digitizing 3D models.

[0040] 2. Generating VR / AR content

[0041] The server generates VR / AR content based on character information converted into a digital format. For example, it 3D models a manga character and adds movement and sound. In this generation process, algorithms for controlling character movements in real time and interactive scenarios to respond to user movements are designed.

[0042] 3. Content distribution

[0043] The server distributes the generated VR / AR content to the corresponding platform. This platform includes dedicated applications, online platforms, and game consoles. The server distributes the content in a format suitable for each platform and adjusts it to ensure smooth access for users.

[0044] 4. User Access Management

[0045] The server verifies the user's access rights to the requested content. It checks whether the user has the appropriate permissions and grants access to the content if they do. It records logs of when access is granted and uses them for later analysis and operation. This makes it possible to understand user behavior history and popular content.

[0046] Specific example

[0047] Specific examples are given below.

[0048] 1. Specific examples of character data collection and digitization

[0049] The server receives data of a "famous cartoon character" from a content provider. This data includes a full-body image of the character, audio clips, and character design documents. The server then converts this data into a digital format and creates a 3D model.

[0050] 2. Specific examples of VR / AR content generation

[0051] The server generates VR content of "famous manga characters" based on the created 3D models. Motion capture data is used to make the characters' movements appear natural. Furthermore, voice recognition functionality is incorporated to allow users to interact with the characters.

[0052] 3. Specific examples of content distribution

[0053] The server uploads the generated VR content of "famous cartoon characters" to a dedicated application. Users can download this content through the application and enjoy it on their VR headsets at home.

[0054] 4. Specific Examples of User Access Management

[0055] Users attempt to access VR content featuring "famous manga characters" through a dedicated application. The server verifies whether the user has access rights and, if access is permitted, records the user's access log. This ensures that users can enjoy the content safely and smoothly.

[0056] This invention realizes a system that efficiently and effectively handles everything from the digitization of character data to the generation and distribution of VR / AR content, as well as user access management. This will enable the development of character businesses in the Japanese entertainment sector on a new platform and make them available to users worldwide.

[0057] The following describes the processing flow.

[0058] Step 1:

[0059] The server collects character data from content providers. This character data includes a wide variety of information, such as images, audio, text, and 3D models. For example, various illustrations of manga characters and animation direction materials are provided.

[0060] Step 2:

[0061] The server converts the collected character data into a digital format. For example, it might scan hand-drawn illustrations into high-resolution image files or create character models using 3D modeling software.

[0062] Step 3:

[0063] The server generates VR (virtual reality) or AR (augmented reality) content based on character data converted into a digital format. For example, it might add movement to 3D models or implement voice recognition functionality. This creates content that allows users to interact and communicate with the characters.

[0064] Step 4:

[0065] The server distributes the generated VR / AR content to the specified platform. This distribution destination can be a dedicated application, an online platform, or a game console. For example, it might package the content into a compressed file for a dedicated application and upload it.

[0066] Step 5:

[0067] The server verifies the user's access rights. When a user attempts to access content, the server checks whether they have the appropriate permissions based on their authentication information. For example, it checks whether they are a paid member or have purchased a specific license.

[0068] Step 6:

[0069] If a user has the appropriate access rights, the server will grant them access to the content. The server records logs of users who have been granted access for later analysis and operational management. For example, it records user ID, access date and time, and the type of content accessed.

[0070] Step 7:

[0071] Users can access permitted content. They can enjoy interactive experiences using VR headsets and AR-enabled devices. For example, they can wear a VR headset to interact with cartoon characters in a virtual space, or use AR devices to make characters appear in the real world.

[0072] The above outlines the specific processing flow in the system of the present invention. This step-by-step processing makes it possible to efficiently manage everything from character data collection to user experience.

[0073] (Example 1)

[0074] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0075] In systems that generate virtual reality (VR) or augmented reality (AR) content based on existing character data, allowing users to access and experience it in real time, there is a need to streamline the content generation process and manage user access more securely and effectively. In particular, there is a lack of systems that can smoothly digitize hand-drawn illustrations and 3D models and design interactive scenarios based on them.

[0076] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0077] In this invention, the server includes means for acquiring content data from a content provider and converting it into a digital format; means for generating content for virtual reality or augmented reality based on the digitally converted content data; means for designing interactive scenarios for the generated content so that users can interact with the content in real time; means for delivering the generated virtual reality or augmented reality content to a corresponding platform; and means for verifying user access rights to the requested content and recording access logs. This makes it possible to consistently and efficiently carry out everything from content generation to delivery and user access management.

[0078] A "content provider" is an organization or individual that possesses character data to be converted into a digital format and provides that data to the server.

[0079] "Content data" refers to information necessary for virtual reality and augmented reality content, such as character images, voices, 3D models, and movements.

[0080] A "digital format" refers to a format of digital data that can be processed by a computer.

[0081] "Virtual reality" is a technology that immerses users in a computer-generated three-dimensional environment.

[0082] Augmented reality is a technology that overlays digital information onto images of the real world.

[0083] An "interactive scenario" refers to the design of a story or event that changes dynamically in response to user input and actions.

[0084] A "platform" refers to an application or online system used to deliver virtual reality or augmented reality content.

[0085] "Access rights" refer to the permissions a user has to access specific content.

[0086] An "access log" refers to a record of when a user accesses a system, and includes information such as the date and time and the content accessed.

[0087] "Hand-drawn illustrations" refer to images drawn by hand on paper or other physical media, and include converting them into a digital format.

[0088] A "3D model" refers to a three-dimensional character or object generated by a computer.

[0089] A "dedicated application" refers to specific software intended for the playback or use of virtual reality or augmented reality content.

[0090] An "online platform" refers to a website or cloud system that provides virtual reality or augmented reality content via the internet.

[0091] A "dedicated terminal" refers to a specific hardware device used to experience virtual reality or augmented reality content.

[0092] This invention relates to a system that generates virtual reality (VR) or augmented reality (AR) content based on character data, which users can access in real time. This system consists of three main elements: a server, a terminal, and a user.

[0093] Character data collection and digitization

[0094] The server receives character data from content providers. This character data includes information such as character images, 3D models, voice, and movements. The server converts the received data into a digital format. Specifically, this involves using technologies to digitize hand-drawn illustrations using high-resolution scanners, or technologies to generate 3D models using 3D scanners.

[0095] For example, we can digitize full-body images, audio clips, and character design documents of a "famous manga character" and convert them into 3D models.

[0096] VR / AR content generation

[0097] The server generates VR / AR content based on digitized character information. Specifically, it uses 3D modeling software (e.g., Blender, Maya) to fine-tune the character's 3D model and imports motion capture data to add natural movements. Furthermore, it uses speech recognition software (e.g., Google® Speech-to-Text) to generate the character's voice. The server then uses the Unity engine to create a VR / AR scene using the generated 3D model and voice data.

[0098] For example, you can optimize the polygon count of a character's 3D model in Blender and then use the Unity engine to create a scene where the character walks and talks.

[0099] Content distribution

[0100] The server distributes the generated VR / AR content to the corresponding platform. Specifically, it uploads the content to a dedicated application (e.g., Android® app, iOS app) and uses an API to deliver the content to the user's device in the appropriate format. If the content is updated, the server automatically distributes the new version.

[0101] For example, VR content of a "famous manga character" that has been generated can be uploaded to a dedicated application, allowing users to download this content through the app.

[0102] User access management

[0103] The server verifies the user's access rights to the requested content and records access logs. Specifically, when a user logs into a dedicated application and submits a request for content, the server compares the user's authentication information with the database to check for access rights. If access is permitted, the server provides the user with an access link to the content and records the access log.

[0104] As a concrete example, when a user attempts to access VR content featuring a "famous manga character" using a dedicated application, the server verifies the user's login information, provides an access link if they have access rights, and records an access log.

[0105] Specific examples and prompt statements

[0106] Example 1: The server downloads character data from "http: / / provider.example.com / character1.zip", unzips the ZIP file, converts it to PNG format, and saves it to the database.

[0107] Example 2: The server uses Blender to optimize a 3D model of a "famous anime character" and then creates a VR scene by adding motion capture data in Unity.

[0108] Specific example 3: The server uploads the generated VR content of "famous manga characters" to a dedicated application, and the user downloads it through the application.

[0109] Specific example 4: When a user attempts to access VR content of a "famous manga character" using a dedicated application, the server verifies the user's login information, provides an access link if they have access rights, and records an access log.

[0110] Example of a prompt:

[0111] "Generate VR / AR content based on data of famous manga characters, verify user access rights, and distribute the content."

[0112] This system enables efficient and effective digitization of character data, generation and distribution of VR / AR content, and management of user access. This allows for the development of character-based businesses in the entertainment sector on a new platform, providing engaging content to users worldwide.

[0113] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0114] Step 1:

[0115] Character data collection and digitization

[0116] The server receives character data from content providers. This character data includes image files, audio files, 3D models, etc. The server converts this data into a digital format and stores it in a dedicated database.

[0117] Specifically, the server downloads character data from a specified URL. For example, it downloads a ZIP file from "http: / / provider.example.com / character1.zip", extracts PNG image files and FBX 3D models, and then stores this data in a database.

[0118] input:

[0119] Character data (images, audio, 3D models) from content providers

[0120] Data processing:

[0121] Downloading and extracting ZIP files

[0122] Convert image files to PNG format and 3D models to FBX format.

[0123] output:

[0124] Character data converted to digital format (PNG image files, FBX 3D models, etc.)

[0125] Step 2:

[0126] VR / AR content generation

[0127] The server generates VR / AR content based on digitized character information. Specifically, it uses 3D modeling software (Blender, Maya) to fine-tune the character's 3D model and imports motion capture data to add natural movements to the character. Furthermore, it uses speech recognition software (Google Speech-to-Text) to generate the character's voice. Finally, the server uses the Unity engine to create a VR / AR scene using the generated 3D model and voice data.

[0128] Specifically, the process involves using Blender to optimize the polygon count of the character's 3D model and then using the Unity engine to create scenes where the character walks and talks.

[0129] input:

[0130] Digitized character data (PNG image files, FBX 3D models, etc.)

[0131] Data processing:

[0132] Detailed adjustments in 3D modeling software

[0133] Importing motion capture data

[0134] Generate audio data using speech recognition software

[0135] Creating VR / AR scenes with the Unity engine

[0136] output:

[0137] VR / AR content (scenes integrating 3D models, motion, and sound)

[0138] Step 3:

[0139] Content distribution

[0140] The server distributes the generated VR / AR content to the corresponding platform. Specifically, it uploads the content to a dedicated application (Android app, iOS app) and uses an API to deliver the content to the user's device in the appropriate format. If the content is updated, the server automatically distributes the new version.

[0141] Specifically, the generated VR content is uploaded to "https: / / app.example.com / updates," a dedicated application checks for the latest version of the content, and a download notification is displayed to the user.

[0142] input:

[0143] Generated VR / AR content (a scene integrating 3D models, motion, and sound)

[0144] Data processing:

[0145] Upload content

[0146] Delivery adjustment using API

[0147] output:

[0148] Content delivered to the user's device

[0149] Step 4:

[0150] User access management

[0151] The server verifies the user's access rights to the requested content and records access logs. Specifically, when a user logs into a dedicated application and submits a request for content, the server compares the user's authentication information with a database to check for access rights. If access rights are granted, the server provides an access link to the content and records access logs.

[0152] Specifically, the user logs in to a dedicated application to access VR content featuring "famous manga characters." The server verifies the user's ID and password, confirms access rights, provides an access link, and logs the access time and content.

[0153] input:

[0154] User login information (ID, password)

[0155] Content Request

[0156] Data processing:

[0157] Verification of user authentication information

[0158] Access rights verification

[0159] Access log recording

[0160] output:

[0161] Access links to content for which access rights have been verified.

[0162] Recorded access logs

[0163] (Application Example 1)

[0164] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0165] Traditionally, systems that provide character data as virtual reality (VR) or augmented reality (AR) content have struggled to efficiently integrate multiple functions such as data digitization, content generation, content distribution, and user access management. Furthermore, there was a lack of technology to effectively display characters in the user's real-world environment and provide an interactive experience. As a result, it was difficult for users to obtain a consistent AR experience.

[0166] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0167] In this invention, the server includes means for generating character data converted into a digital format, means for generating content for virtual reality or augmented reality based on the character information converted into a digital format, means for distributing the generated content to a corresponding platform, means for verifying access rights to content requested by the user and recording access logs, means for acquiring a camera feed from a smart device using augmented reality software and displaying a 3D model on the acquired camera feed, and means for playing audio that interacts with the 3D model. This enables the user to naturally display the character in the real environment and seamlessly enjoy an interactive experience.

[0168] A "digital format" refers to a form that can be stored and handled as digital data.

[0169] "Character data" refers to information about characters that appear in manga, anime, games, etc., including images, models, voices, and movements.

[0170] "Virtual reality (VR)" refers to a technology that provides interactive experiences within a virtual 3D space created using computer technology.

[0171] Augmented reality (AR) refers to a technology that overlays virtual information onto the real, physical environment.

[0172] "Content" refers to information and media (such as videos, audio, and 3D models) that users consume.

[0173] A "platform" refers to an environment, infrastructure, or service that provides the foundation for software and hardware to operate.

[0174] "Access rights" refer to permission to operate on or use specific content or services.

[0175] "Access logs" refer to data that records the history of how users have accessed a system or content.

[0176] "Augmented reality software" refers to software that overlays digital information onto the real world.

[0177] "Smart devices" refer to mobile terminals, eyeglasses, and wearable devices that have internet connectivity and high processing power.

[0178] "Camera feed" refers to real-time video footage captured by a camera.

[0179] A "3D model" refers to digital data that simulates a three-dimensional object, created using a computer.

[0180] "Audio playback means" refers to devices or functions for playing audio data through devices such as speakers or headphones.

[0181] "Interaction" refers to the two-way exchange that takes place between a user and a digital system or content.

[0182] The system realizing this invention has the function of collecting and generating character data converted into a digital format, generating virtual reality (VR) or augmented reality (AR) content based on that data, and distributing it to users. The specific configuration and operation of this system are described below.

[0183] 1. Character data collection and digitization

[0184] The server first receives character data from content providers (such as publishing companies or game companies). This data includes information such as character images, 3D models, audio clips, and motion data. This character data is then converted into a digital format using scanning and modeling technologies. This conversion enables the creation of high-quality digital content.

[0185] 2. Generating VR / AR content

[0186] The server generates VR / AR content based on character data converted into a digital format. Here, for example, 3D modeling and motion capture technology are used to create interactive characters with natural movements. Voice recognition functionality is also integrated to create characters that respond to user voices.

[0187] 3. Content distribution

[0188] The server distributes the generated VR / AR content to the corresponding platform. This platform includes dedicated applications, online platforms, and game consoles. By using smart devices (e.g., smartphones and smart glasses), users can have an interactive experience.

[0189] 4. Realization of Augmented Reality

[0190] The server uses augmented reality software to retrieve the camera feed from the smart device. A 3D model of the character is then overlaid onto the retrieved camera feed. For example, if the user is wearing smart glasses, the character will appear in the real-world scene, allowing for interaction with its movements and sounds.

[0191] 5. User Access Management

[0192] The server checks the user's access rights to the requested content and logs the access if it is granted. This log is used for later analysis and service improvement.

[0193] Explanation of specific examples

[0194] For example, suppose a user wants to enjoy an AR experience with a famous character. When the user opens a dedicated application and puts on smart glasses, the server delivers a 3D model and voice data of the character. As the user walks around in the real world, the character appears in that location and speaks and acts in response to the user's voice. Such an experience makes it feel as if the character is actually present in reality.

[0195] Example of a prompt

[0196] Design an application that provides an AR experience using a 3D model and voice data of Naruto. The application will allow users to interact with the character in real time using smart glasses. The character's movements and voice should be natural and interactive.

[0197] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0198] Step 1:

[0199] The server retrieves character data (images, 3D models, audio clips, motion data, etc.) from content providers. This data is accessed directly from the content provider's database or received as a provided file. The retrieved data is temporarily stored on the server. The input is character data, and the output is raw character data stored on the server.

[0200] Step 2:

[0201] The server converts the acquired character data into a digital format. For example, it can digitize hand-drawn illustrations using scanning technology or convert existing 2D images into 3D models. This conversion process uses specialized software (e.g., Adobe Photoshop, Blender, etc.). The input is raw character data, and the output is character data converted into a digital format.

[0202] Step 3:

[0203] The server generates VR / AR content based on character data converted into a digital format. This process uses modeling software to recreate natural character movements and incorporates voice recognition and interactive features. For example, motion capture technology is used to control character movements in real time. The input is character data in a digital format, and the output is content optimized for VR / AR.

[0204] Step 4:

[0205] The server distributes the generated VR / AR content to the corresponding platform. This platform includes dedicated applications, online platforms, and apps for smart devices, and the content needs to be converted to an appropriate format for each platform. The input is the VR / AR content, and the output is the content converted to a format suitable for the platform.

[0206] Step 5:

[0207] The device acquires the camera feed from a smart device. A 3D character model received from a server is overlaid on the acquired real-time video data. Specifically, augmented reality software is used to position the character according to real-world coordinates and enable interactive actions. The input is the camera feed and the 3D character model, and the output is the video displayed in augmented reality.

[0208] Step 6:

[0209] The device receives voice input from the user and converts it to text using a speech recognition engine (e.g., Google Speech-to-Text API). This text is then analyzed to determine how the character should react. The input is the user's voice, and the output is the text data and the character's reaction.

[0210] Step 7:

[0211] The server manages user access rights and verifies access rights to requested content. If access is granted, it logs that information and grants the user access to the content. The input is the user's request information, and the output is log data regarding access rights and the access permissions.

[0212] Step 8:

[0213] The server analyzes user access logs to understand popular content and user behavior patterns. This analysis is used to improve services and develop new content. The input is access log data, and the output is the analysis results.

[0214] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0215] This invention is a system that converts character data into a digital format, generates virtual reality (VR) or augmented reality (AR) content, and delivers that content to a platform. Furthermore, it incorporates an emotion engine that recognizes user emotions and dynamically changes the content accordingly.

[0216] 1. Character data collection and digitization

[0217] The server first collects character data from content providers (such as publishing companies and game companies). This data includes images, voices, text, and 3D models of each character. The collected data is then converted into a digital format. This process utilizes high-resolution image scanning and 3D modeling software.

[0218] 2. Generating VR / AR content

[0219] The server generates VR and AR content based on digitized character data. This content can include 3D models, audio, and movement. It also designs specific algorithms and scenarios to allow users to interact with the characters.

[0220] 3. Integrating an emotion engine

[0221] The server incorporates an emotion engine that recognizes the user's emotions in real time. The emotion engine uses cameras and microphones to analyze the user's facial expressions, voice, and gestures to determine the user's emotional state. For example, if the user is smiling, it determines they are "happy," and if they are frowning, it determines they are "sad."

[0222] 4. Dynamic content changes based on emotions

[0223] The server dynamically changes the VR / AR content based on the user's emotions, as recognized by the emotion engine. For example, if the user is perceived as "happy," the character will perform more friendly actions. Conversely, if the user is perceived as "sad," the character will perform comforting actions.

[0224] 5. Content distribution

[0225] The server distributes the generated VR / AR content to the appropriate platform (such as a dedicated application, online platform, or game console). It delivers the content in a format suitable for each platform, ensuring easy access for users.

[0226] 6. User Access Management

[0227] The server verifies the user's access rights to the requested content. Users whose access rights are verified are granted access to the content, and an access log is recorded. This log includes the user ID, access date and time, and the type of content accessed.

[0228] Specific example

[0229] 1. Specific examples of character data collection and digitization

[0230] The server receives data for "Popular Character A" from the content provider. This data includes a full-body image, audio clips, and character design documents. The server then converts this data into a digital format and creates a high-resolution 3D model.

[0231] 2. Specific examples of VR / AR content generation

[0232] The server generates VR content for "Popular Character A" based on the created 3D model. Motion capture data is used to give the character realistic movements, and voice recognition functionality is added to implement user interaction.

[0233] 3. Specific Examples of Emotional Engines

[0234] The server activates an emotion engine to analyze the user's facial expressions and voice. For example, if the user is smiling, it recognizes that they are "happy," and if they are frowning, it recognizes that they are "sad."

[0235] 4. Specific examples of dynamic content changes based on emotions

[0236] The server configures the VR content's characters to perform friendly actions and speak in a way that indicates the user is "happy." Conversely, if the user is perceived as "sad," the characters' actions are adjusted to offer comforting words.

[0237] 5. Specific examples of content distribution

[0238] The server uploads the generated VR content of "Popular Character A" to a dedicated application. Users can access this content through the dedicated application and enjoy the experience on their home VR headset.

[0239] 6. Specific Examples of User Access Management

[0240] A user attempts to access VR content featuring "Popular Character A." The server verifies the user's authentication information and confirms access rights. If access is permitted, the user's access log is recorded, and their usage history for the content is managed.

[0241] This enables the system of the present invention to efficiently generate advanced character content and provide interactive experiences that respond to user emotions. This system allows character businesses in the Japanese entertainment sector to effectively expand on a new platform, providing compelling experiences to users worldwide.

[0242] The following describes the processing flow.

[0243] Step 1:

[0244] The server collects character data from content providers. This data includes character images, audio, text, and 3D models. For example, it might receive full-color illustrations of manga characters or audio data of character dialogue.

[0245] Step 2:

[0246] The server converts the received character data into a digital format. This conversion includes scanning the image data into high-resolution electronic files and generating a 3D model from the scanned data. For example, software is used to create 3D models from 2D illustrations.

[0247] Step 3:

[0248] The server generates VR and AR content based on character data converted into a digital format. Specifically, it adds movement to 3D models and implements speech recognition and generation functions to enable dialogue. Motion capture technology is used to capture natural character movements.

[0249] Step 4:

[0250] The server activates an emotion engine to analyze the user's facial expressions and voice in real time. It uses cameras and microphones to recognize the user's emotions, and the emotion engine analyzes the acquired data. For example, if the user is smiling, it determines they are "happy," and if they are frowning, it recognizes them as "sad."

[0251] Step 5:

[0252] The server dynamically modifies VR and AR content based on the recognized user's emotions. The character's actions and dialogue change according to the emotion engine's results. For example, if the user is judged to be "happy," the character will respond with a smile. If the user is judged to be "sad," the character will perform comforting actions or speak comforting words.

[0253] Step 6:

[0254] The server distributes the generated VR / AR content to the appropriate platform. It distributes the content to dedicated applications, online platforms, game consoles, etc., making it accessible to users. For example, it uploads VR content to a dedicated app, allowing users to download it.

[0255] Step 7:

[0256] A user attempts to access VR / AR content. The server checks the user's access rights and authenticates whether they have the appropriate permissions. If the user has the necessary permissions, access to the content is permitted.

[0257] Step 8:

[0258] The server records user access logs. It records which content users accessed and when, and uses this information for later analysis and operational management. For example, it stores user IDs, access dates and times, and the type of content accessed in a database.

[0259] Step 9:

[0260] Users can access permitted content. They can enjoy interactive experiences using VR headsets and AR-enabled devices. For example, they can wear a VR headset and interact with characters in a virtual space. They can also use AR devices to make characters appear in the real world.

[0261] The above outlines the specific processing flow in the system of the present invention that incorporates an emotion engine. This step-by-step processing makes it possible to manage everything from character data collection to user experience efficiently and interactively.

[0262] (Example 2)

[0263] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0264] In recent years, virtual reality (VR) and augmented reality (AR) technologies have developed rapidly, and many users are seeking interactive and engaging content. However, current systems struggle to recognize user emotions in real time and dynamically change content accordingly. As a result, the quality of the user experience is limited, and in some scenarios, there is a lack of immersion. Furthermore, the effort and resources required to support multiple platforms also remain challenges.

[0265] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0266] In this invention, the server includes means for generating character data converted into a digital format, means for generating content for virtual reality or augmented reality based on the character information converted into a digital format, means for distributing the generated content to a corresponding platform, means for verifying access rights to content requested by the user and recording access logs, an emotion recognition engine that recognizes the user's emotions in real time using a camera and microphone, and means for dynamically changing the content for virtual reality or augmented reality based on the user's emotions. This enables real-time content adjustment in response to the user's emotions, resulting in a more immersive and interactive experience.

[0267] A "digital format" is a format in which physical analog data is converted into digital data that can be used by a computer.

[0268] "Character data" refers to information about characters used in virtual reality and augmented reality content, and includes images, audio, text, 3D models, and more.

[0269] Virtual reality (VR) is a technology that allows humans to experience virtual spaces and objects created using computer technology through their five senses.

[0270] Augmented reality (AR) is a technology that overlays digital information onto real-world scenery, integrating computer-generated visual elements into the real world for display.

[0271] A "content provider" refers to a company or organization that generates and distributes digital content, and this includes publishing companies, game companies, and others.

[0272] A "platform" refers to the underlying environment or services for running and providing software and content, and includes dedicated applications, online platforms, and game consoles.

[0273] "Access rights" refer to the right of a user to access specific digital content or resources.

[0274] An "access log" is data that records the history of access to a system or database, and includes information such as the user's ID, the date and time of access, and the type of content accessed.

[0275] An "emotion recognition engine" is a software engine that uses sensors such as cameras and microphones to analyze a user's facial expressions, voice, and gestures, and recognizes their emotional state in real time.

[0276] "Dynamic modification" refers to changing the operation or output of a system in real time or on an ad-hoc basis in response to specific conditions or data.

[0277] This invention is a system that converts character data into a digital format, generates virtual reality (VR) or augmented reality (AR) content, and delivers that content to a platform. Furthermore, it incorporates an emotion recognition engine that recognizes the user's emotions and dynamically changes the content accordingly.

[0278] Character data collection and digitization

[0279] The server first collects character data from content providers (such as publishing companies and game companies). The collected data includes images, voices, text, and 3D models for each character. Next, this data is converted into digital formats using high-resolution image scanning and 3D modeling software. Specifically, image data is edited using Adobe Photoshop, converted to high-resolution images, and 3D models are generated using Autodesk Maya.

[0280] VR / AR content generation

[0281] The server generates VR and AR content based on digitized character data. This content includes 3D models, voices, and movements. Design interaction scenarios and interaction algorithms so that users can interact with the character interactively. For specific development, use game engines such as Unity and Unreal Engine, load motion capture data, and use the Google Cloud Speech-to-Text API for speech recognition.

[0282] Integration of emotion recognition engine

[0283] The server incorporates an emotion recognition engine that recognizes the user's emotions in real time using a camera and a microphone. This engine analyzes the user's expressions using face recognition libraries such as OpenCV and Dlib, and recognizes emotions from voice data using Microsoft (registered trademark) Azure (registered trademark)'s Emotion API.

[0284] As a specific example, when the user connects to the system, the camera captures the user's face and analyzes the expression using OpenCV. The voice data from the microphone is processed through the Emotion API, and the user's emotions are determined in real time.

[0285] Dynamic change of content according to emotions

[0286] The server dynamically changes the VR / AR content based on the user's emotions recognized by the emotion recognition engine. For example, when the user is recognized as "happy", the character performs more friendly actions, and when the user is recognized as "sad", comforting actions are executed. This process is implemented using action scripts (for example, Python) to call appropriate animations and lines based on specific emotional states.

[0287] Distribution of content

[0288] The server delivers the generated VR / AR content to the appropriate platform. It delivers the content in a format compatible with dedicated applications, online platforms, game consoles, etc., making it easily accessible to users. Specifically, it uses a system that uploads content to an AWS® S3 bucket and delivers it to users worldwide with low latency using a CDN (Content Delivery Network).

[0289] User access management

[0290] The server verifies the user's access rights to the requested content. Users who are granted access are allowed to access the content, and an access log is recorded. This log includes information such as the user ID, access date and time, and the type of content accessed. Authentication is performed using OAuth or JWT (JSON Web Token), and the access log is recorded in a database such as MySQL® or MongoDB.

[0291] Specific example

[0292] 1. Specific examples of character data collection and digitization

[0293] The server downloads full-body images, audio clips, and setting materials of "Popular Character A" via the publishing company's API, digitizes the image data using Adobe Photoshop, and creates a high-resolution 3D model using Autodesk Maya.

[0294] 2. Specific examples of VR / AR content generation

[0295] The server uses the Unity engine to add movement to a 3D model of "Popular Character A," implements speech recognition functionality using the Google Cloud Speech-to-Text API, and designs user interaction scenarios.

[0296] 3. Specific Examples of Emotion Recognition Engines

[0297] The server uses OpenCV to capture the user's facial expressions in real time and the Microsoft Azure Emotion API to recognize emotions from the audio data. It determines the emotional state, such as "happy" if the user is smiling and "sad" if they are frowning.

[0298] 4. Specific examples of dynamic content changes based on emotions

[0299] If the server determines that the user is "happy," the character will smile and speak in a friendly manner; if it determines that the user is "sad," the character will offer words of comfort.

[0300] 5. Specific examples of content distribution

[0301] The server uploads VR content of "Popular Character A" generated in an AWS S3 bucket, and users can access the content through a dedicated application and experience it using their home VR headset.

[0302] 6. Specific Examples of User Access Management

[0303] The user logs into the app to access VR content featuring "Popular Character A," and the server performs user authentication using OAuth. If authentication is successful, the user's access log is recorded in a MySQL database.

[0304] This system enables real-time content adjustments based on user emotions, resulting in a more immersive and interactive experience.

[0305] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0306] Step 1: Collection of Character Data

[0307] The server collects character data from content providers. The input is character data (images, voices, texts, 3D models) provided by publishing companies or game companies. The output is stored in the server as the collected data. Specifically, the server sends an API request to the content provider to obtain the data and saves it in its own storage.

[0308] Step 2: Digitization of Character Data

[0309] The server converts the collected character data into a digital format. The input is the collected analog data or low-resolution digital data. The output is character data formatted as high-resolution digital data. Specifically, Adobe Photoshop is used to convert image data to high resolution, and Autodesk Maya is used to create 3D models.

[0310] Step 3: Generation of VR / AR Content

[0311] The server generates VR or AR content based on the digitized character data. The input is the digitized 3D model and voice data. The output is interactive VR / AR content. Specific operations include using Unity or Unreal Engine to add movement to the 3D model and implementing a voice recognition function using the Google Cloud Speech-to-Text API.

[0312] Step 4: Activation of the Emotion Recognition Engine

[0313] The server activates an emotion recognition engine that recognizes the user's emotions in real time. The input consists of facial video and audio data sent from the user's terminal. The output is the emotion recognition result. Specifically, the server uses OpenCV and Dlib to analyze video captured from the camera, and the Microsoft Azure Emotion API to analyze audio data.

[0314] Step 5: Dynamically change content based on emotions

[0315] The server dynamically modifies VR / AR content based on the user's emotions recognized by the emotion recognition engine. The input is the emotion recognition result. The output is VR / AR content optimized according to the user's emotions. Specifically, the server uses an action script written in Python to call appropriate animations and dialogue in real time based on a specific emotional state.

[0316] Step 6: Content Distribution

[0317] The server delivers the generated VR / AR content to the corresponding platform. The input is the completed VR / AR content. The output is the content provided in a state accessible to the user. Specifically, the content is uploaded to an AWS S3 bucket and delivered to users worldwide with low latency using a CDN.

[0318] Step 7: Managing User Access

[0319] The server verifies the user's access rights to the requested content and records access logs. The input is the user's authentication information. The output is the content the user can access after authentication, and log data representing the access history. Specifically, authentication is performed using OAuth or JWT, and the results are recorded in a MySQL or MongoDB database.

[0320] This processing flow allows for the rapid and efficient delivery of interactive VR / AR experiences that respond to the user's emotions.

[0321] (Application Example 2)

[0322] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0323] Traditional virtual reality (VR) and augmented reality (AR) content systems face the challenge of providing interactive experiences that respond to user emotional changes. Furthermore, users are often limited to certain patterns of actions and scenarios when enjoying the content, resulting in a lack of realism and immersion. Additionally, there is a lack of technology to dynamically modify content based on real-time user feedback.

[0324] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for generating character data converted into a digital format, means for generating content for virtual reality or augmented reality based on the character information converted into a digital format, means for distributing the generated content to a corresponding platform, means for confirming access rights to the content requested by the user and recording access logs, means for analyzing the user's facial expressions and voice to recognize emotions, and means for dynamically changing the content based on the recognized emotions. This makes it possible to provide an interactive VR / AR experience that responds to the user's emotional state.

[0325] A "digital format" is a format in which analog data has been converted into a format that can be processed on a computer.

[0326] "Character data" refers to information such as images, audio, 3D models, and text related to a specific character.

[0327] "Virtual reality" is a technology that allows users to experience a three-dimensional artificial environment created using computer technology, as if it were reality, through their sight and hearing.

[0328] Augmented reality is a technology that overlays virtual information onto the real world, allowing users to view both the real world and digital information simultaneously.

[0329] A "platform" refers to the underlying software and hardware environment on which applications and content operate.

[0330] "Access rights" refer to the right of a user to access specific content or information, and authentication determines whether a user is entitled to access that information.

[0331] An "access log" is a record of a user's access history kept by the system, and includes information such as the user ID, the date and time of access, and the type of content accessed.

[0332] "Facial expression" refers to emotions and reactions that are expressed through the movement of the facial structure.

[0333] "Sound" refers to sounds that are transmitted as sound waves and can be heard by the human ear, and mainly includes human speech.

[0334] "Emotion" refers to the psychological response or state that a user experiences in response to a particular situation or stimulus.

[0335] "Dynamic modification" means instantly changing content and information in response to real-time changes in circumstances.

[0336] "Interactive" refers to a system that interacts with the user and responds immediately to user input.

[0337] This invention relates to a system that converts character data into a digital format, generates virtual reality (VR) or augmented reality (AR) content, and incorporates an emotion engine that recognizes user emotions and dynamically changes the content. This system is implemented using the following steps, hardware, and software.

[0338] 1. Collection and digitization of character data

[0339] The server first collects character data from content providers. This data includes images, audio, text, and 3D models of each character. This collected data is then converted into a digital format and stored. This process utilizes high-resolution image scanning and 3D modeling software.

[0340] 2. Generating VR / AR content

[0341] The server generates VR and AR content based on character data converted into a digital format. At this stage, elements such as 3D models, voice, and movement are combined. Furthermore, certain algorithms and scenarios are designed to enable interactive dialogue.

[0342] 3. Integrating an emotion engine

[0343] An emotion engine has been introduced to recognize the user's emotions in real time. Using cameras and microphones, it collects data on the user's facial expressions, voice, and gestures, and uses this data to determine their emotional state. For example, if the user is smiling, it will be determined to be "happy," and if they are frowning, it will be determined to be "sad."

[0344] 4. Dynamic content changes based on emotions

[0345] The server dynamically changes the VR / AR content based on the user's emotions, as recognized by the emotion engine. For example, if the user is perceived as "happy," the character will perform more friendly actions. Conversely, if the user is perceived as "sad," the character will perform comforting actions.

[0346] 5. Content distribution

[0347] The generated VR / AR content is delivered to the appropriate platform (such as a dedicated application, online platform, or game console). The content is delivered in a format suitable for each platform, ensuring easy access for users.

[0348] 6. User Access Management

[0349] The server checks the user's access rights to the requested content and records access logs as needed. The access logs include the user ID, access date and time, and the type of content accessed.

[0350] Hardware and software to use

[0351] This system uses the following hardware and software.

[0352] Hardware: Webcam, VR headset, smartphone

[0353] Software: Python, OpenCV, Dlib, Pyttsx3, 3D modeling software

[0354] Specific example

[0355] For example, in a system applying the present invention, suppose a user puts on a VR headset and begins interacting with "Popular Character A". If the user smiles, the animation changes to the character speaking in a friendly manner. Conversely, if the user shows a sad expression, the scenario changes to one in which the character offers words of comfort.

[0356] Example of a prompt

[0357] Please create a VR character that responds in a friendly manner when the user is smiling and in a comforting manner when the user is sad.

[0358] The necessary data is to read the user's emotions from their facial expressions and change the behavior based on that.

[0359] Use Python's OpenCV and Dlib libraries to perform emotion recognition, and the Python library pyttsx3 to provide voice feedback.

[0360] Thus, this invention provides an interactive VR / AR experience based on the user's emotions, enabling a more personalized and engaging experience for the user.

[0361] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0362] Step 1:

[0363] This system collects and digitizes character data. The server collects data such as images, audio, text, and 3D models related to characters from content providers. Input is character data, and output is character data converted to a digital format. High-resolution image scanning and 3D modeling software are used to convert the data to a digital format and save it.

[0364] Step 2:

[0365] This system generates VR / AR content. The server uses character data converted to a digital format to create VR and AR content by combining 3D models, audio, and movement. Input is character data in digital format, and output is VR / AR content. Specific algorithms and scenarios are also designed to enable interactive dialogue.

[0366] Step 3:

[0367] An emotion engine will be implemented. The server will use the emotion engine to analyze the user's facial expressions and voice to recognize their emotions. The input will be user facial expression and voice data collected by cameras and microphones, and the output will be the recognized emotional state. Data will be collected using cameras and microphones, and the user's emotions will be determined using software such as OpenCV or Dlib.

[0368] Step 4:

[0369] The content is dynamically changed according to emotions. The server dynamically changes the VR / AR content based on the user's emotions recognized by the emotion engine. The input is the recognized emotional state, and the output is the dynamically changed VR / AR content. For example, if the user is recognized as "happy," the character is set to perform friendly actions.

[0370] Step 5:

[0371] The content is delivered to the platform. The server uploads the generated VR / AR content to a dedicated application or online platform. The input is the generated VR / AR content, and the output is the content accessible on the platform. The content is delivered in an appropriate format, making it easily accessible to users.

[0372] Step 6:

[0373] This system manages user access. The server verifies user access rights to requested content and records access logs. Inputs include user requests and authentication information, while outputs include access rights verification and access logs. Users whose access rights have been verified are granted access to the content, and the system records user ID, access date and time, and the type of content accessed.

[0374] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0375] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0376] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0377] [Second Embodiment]

[0378] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0379] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0380] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0381] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0382] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0383] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0384] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0385] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0386] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0387] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0388] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0389] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0390] The system for carrying out the present invention will now be described. This system consists of the following main components.

[0391] 1. Character data collection and digitization

[0392] The server first receives character data from content providers (e.g., manga and anime publishers, game companies, etc.). This includes information such as character images, models, voices, and movements. The received character data is then converted into a digital format. This conversion process uses technologies such as scanning to digitize hand-drawn illustrations or digitizing 3D models.

[0393] 2. Generating VR / AR content

[0394] The server generates VR / AR content based on character information converted into a digital format. For example, it 3D models a manga character and adds movement and sound. In this generation process, algorithms for controlling character movements in real time and interactive scenarios to respond to user movements are designed.

[0395] 3. Content distribution

[0396] The server distributes the generated VR / AR content to the corresponding platform. This platform includes dedicated applications, online platforms, and game consoles. The server distributes the content in a format suitable for each platform and adjusts it to ensure smooth access for users.

[0397] 4. User Access Management

[0398] The server verifies the user's access rights to the requested content. It checks whether the user has the appropriate permissions and grants access to the content if they do. It records logs of when access is granted and uses them for later analysis and operation. This makes it possible to understand user behavior history and popular content.

[0399] Specific example

[0400] Specific examples are given below.

[0401] 1. Specific examples of character data collection and digitization

[0402] The server receives data of a "famous cartoon character" from a content provider. This data includes a full-body image of the character, audio clips, and character design documents. The server then converts this data into a digital format and creates a 3D model.

[0403] 2. Specific examples of VR / AR content generation

[0404] The server generates VR content of "famous manga characters" based on the created 3D models. Motion capture data is used to make the characters' movements appear natural. Furthermore, voice recognition functionality is incorporated to allow users to interact with the characters.

[0405] 3. Specific examples of content distribution

[0406] The server uploads the generated VR content of "famous cartoon characters" to a dedicated application. Users can download this content through the application and enjoy it on their VR headsets at home.

[0407] 4. Specific Examples of User Access Management

[0408] Users attempt to access VR content featuring "famous manga characters" through a dedicated application. The server verifies whether the user has access rights and, if access is permitted, records the user's access log. This ensures that users can enjoy the content safely and smoothly.

[0409] This invention realizes a system that efficiently and effectively handles everything from the digitization of character data to the generation and distribution of VR / AR content, as well as user access management. This will enable the development of character businesses in the Japanese entertainment sector on a new platform and make them available to users worldwide.

[0410] The following describes the processing flow.

[0411] Step 1:

[0412] The server collects character data from content providers. This character data includes a wide variety of information, such as images, audio, text, and 3D models. For example, various illustrations of manga characters and animation direction materials are provided.

[0413] Step 2:

[0414] The server converts the collected character data into a digital format. For example, it might scan hand-drawn illustrations into high-resolution image files or create character models using 3D modeling software.

[0415] Step 3:

[0416] The server generates VR (virtual reality) or AR (augmented reality) content based on character data converted into a digital format. For example, it might add movement to 3D models or implement voice recognition functionality. This creates content that allows users to interact and communicate with the characters.

[0417] Step 4:

[0418] The server distributes the generated VR / AR content to the specified platform. This distribution destination can be a dedicated application, an online platform, or a game console. For example, it might package the content into a compressed file for a dedicated application and upload it.

[0419] Step 5:

[0420] The server verifies the user's access rights. When a user attempts to access content, the server checks whether they have the appropriate permissions based on their authentication information. For example, it checks whether they are a paid member or have purchased a specific license.

[0421] Step 6:

[0422] If a user has the appropriate access rights, the server will grant them access to the content. The server records logs of users who have been granted access for later analysis and operational management. For example, it records user ID, access date and time, and the type of content accessed.

[0423] Step 7:

[0424] Users can access permitted content. They can enjoy interactive experiences using VR headsets and AR-enabled devices. For example, they can wear a VR headset to interact with cartoon characters in a virtual space, or use AR devices to make characters appear in the real world.

[0425] The above outlines the specific processing flow in the system of the present invention. This step-by-step processing makes it possible to efficiently manage everything from character data collection to user experience.

[0426] (Example 1)

[0427] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0428] In systems that generate virtual reality (VR) or augmented reality (AR) content based on existing character data, allowing users to access and experience it in real time, there is a need to streamline the content generation process and manage user access more securely and effectively. In particular, there is a lack of systems that can smoothly digitize hand-drawn illustrations and 3D models and design interactive scenarios based on them.

[0429] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0430] In this invention, the server includes means for acquiring content data from a content provider and converting it into a digital format; means for generating content for virtual reality or augmented reality based on the digitally converted content data; means for designing interactive scenarios for the generated content so that users can interact with the content in real time; means for delivering the generated virtual reality or augmented reality content to a corresponding platform; and means for verifying user access rights to the requested content and recording access logs. This makes it possible to consistently and efficiently carry out everything from content generation to delivery and user access management.

[0431] A "content provider" is an organization or individual that possesses character data to be converted into a digital format and provides that data to the server.

[0432] "Content data" refers to information necessary for virtual reality and augmented reality content, such as character images, voices, 3D models, and movements.

[0433] A "digital format" refers to a format of digital data that can be processed by a computer.

[0434] "Virtual reality" is a technology that immerses users in a computer-generated three-dimensional environment.

[0435] Augmented reality is a technology that overlays digital information onto images of the real world.

[0436] An "interactive scenario" refers to the design of a story or event that changes dynamically in response to user input and actions.

[0437] A "platform" refers to an application or online system used to deliver virtual reality or augmented reality content.

[0438] "Access rights" refer to the permissions a user has to access specific content.

[0439] An "access log" refers to a record of when a user accesses a system, and includes information such as the date and time and the content accessed.

[0440] "Hand-drawn illustrations" refer to images drawn by hand on paper or other physical media, and include converting them into a digital format.

[0441] A "3D model" refers to a three-dimensional character or object generated by a computer.

[0442] A "dedicated application" refers to specific software intended for the playback or use of virtual reality or augmented reality content.

[0443] An "online platform" refers to a website or cloud system that provides virtual reality or augmented reality content via the internet.

[0444] A "dedicated terminal" refers to a specific hardware device used to experience virtual reality or augmented reality content.

[0445] This invention relates to a system that generates virtual reality (VR) or augmented reality (AR) content based on character data, which users can access in real time. This system consists of three main elements: a server, a terminal, and a user.

[0446] Character data collection and digitization

[0447] The server receives character data from content providers. This character data includes information such as character images, 3D models, voice, and movements. The server converts the received data into a digital format. Specifically, this involves using technologies to digitize hand-drawn illustrations using high-resolution scanners, or technologies to generate 3D models using 3D scanners.

[0448] For example, we can digitize full-body images, audio clips, and character design documents of a "famous manga character" and convert them into 3D models.

[0449] VR / AR content generation

[0450] The server generates VR / AR content based on digitized character information. Specifically, it uses 3D modeling software (e.g., Blender, Maya) to fine-tune the character's 3D model and imports motion capture data to add natural movements. Furthermore, it uses speech recognition software (e.g., Google Speech-to-Text) to generate the character's voice. The server then uses the Unity engine to create a VR / AR scene using the generated 3D model and voice data.

[0451] For example, you can optimize the polygon count of a character's 3D model in Blender and then use the Unity engine to create a scene where the character walks and talks.

[0452] Content distribution

[0453] The server distributes the generated VR / AR content to the corresponding platform. Specifically, it uploads the content to a dedicated application (e.g., Android app, iOS app) and uses an API to deliver the content to the user's device in the appropriate format. If the content is updated, the server automatically distributes the new version.

[0454] For example, VR content of a "famous manga character" that has been generated can be uploaded to a dedicated application, allowing users to download this content through the app.

[0455] User access management

[0456] The server verifies the user's access rights to the requested content and records access logs. Specifically, when a user logs into a dedicated application and submits a request for content, the server compares the user's authentication information with the database to check for access rights. If access is permitted, the server provides the user with an access link to the content and records the access log.

[0457] As a concrete example, when a user attempts to access VR content featuring a "famous manga character" using a dedicated application, the server verifies the user's login information, provides an access link if they have access rights, and records an access log.

[0458] Specific examples and prompt statements

[0459] Example 1: The server downloads character data from "http: / / provider.example.com / character1.zip", unzips the ZIP file, converts it to PNG format, and saves it to the database.

[0460] Example 2: The server uses Blender to optimize a 3D model of a "famous anime character" and then creates a VR scene by adding motion capture data in Unity.

[0461] Specific example 3: The server uploads the generated VR content of "famous manga characters" to a dedicated application, and the user downloads it through the application.

[0462] Specific example 4: When a user attempts to access VR content of a "famous manga character" using a dedicated application, the server verifies the user's login information, provides an access link if they have access rights, and records an access log.

[0463] Example of a prompt:

[0464] "Generate VR / AR content based on data of famous manga characters, verify user access rights, and distribute the content."

[0465] This system enables efficient and effective digitization of character data, generation and distribution of VR / AR content, and management of user access. This allows for the development of character-based businesses in the entertainment sector on a new platform, providing engaging content to users worldwide.

[0466] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0467] Step 1:

[0468] Character data collection and digitization

[0469] The server receives character data from content providers. This character data includes image files, audio files, 3D models, etc. The server converts this data into a digital format and stores it in a dedicated database.

[0470] Specifically, the server downloads character data from a specified URL. For example, it downloads a ZIP file from "http: / / provider.example.com / character1.zip", extracts PNG image files and FBX 3D models, and then stores this data in a database.

[0471] input:

[0472] Character data (images, audio, 3D models) from content providers

[0473] Data processing:

[0474] Downloading and extracting ZIP files

[0475] Convert image files to PNG format and 3D models to FBX format.

[0476] output:

[0477] Character data converted to digital format (PNG image files, FBX 3D models, etc.)

[0478] Step 2:

[0479] VR / AR content generation

[0480] The server generates VR / AR content based on digitized character information. Specifically, it uses 3D modeling software (Blender, Maya) to fine-tune the character's 3D model and imports motion capture data to add natural movements to the character. Furthermore, it uses speech recognition software (Google Speech-to-Text) to generate the character's voice. Finally, the server uses the Unity engine to create a VR / AR scene using the generated 3D model and voice data.

[0481] Specifically, the process involves using Blender to optimize the polygon count of the character's 3D model and then using the Unity engine to create scenes where the character walks and talks.

[0482] input:

[0483] Digitized character data (PNG image files, FBX 3D models, etc.)

[0484] Data processing:

[0485] Detailed adjustments in 3D modeling software

[0486] Importing motion capture data

[0487] Generate audio data using speech recognition software

[0488] Creating VR / AR scenes with the Unity engine

[0489] output:

[0490] VR / AR content (scenes integrating 3D models, motion, and sound)

[0491] Step 3:

[0492] Content distribution

[0493] The server distributes the generated VR / AR content to the corresponding platform. Specifically, it uploads the content to a dedicated application (Android app, iOS app) and uses an API to deliver the content to the user's device in the appropriate format. If the content is updated, the server automatically distributes the new version.

[0494] Specifically, the generated VR content is uploaded to "https: / / app.example.com / updates," a dedicated application checks for the latest version of the content, and a download notification is displayed to the user.

[0495] input:

[0496] Generated VR / AR content (a scene integrating 3D models, motion, and sound)

[0497] Data processing:

[0498] Upload content

[0499] Delivery adjustment using API

[0500] output:

[0501] Content delivered to the user's device

[0502] Step 4:

[0503] User access management

[0504] The server verifies the user's access rights to the requested content and records access logs. Specifically, when a user logs into a dedicated application and submits a request for content, the server compares the user's authentication information with a database to check for access rights. If access rights are granted, the server provides an access link to the content and records access logs.

[0505] Specifically, the user logs in to a dedicated application to access VR content featuring "famous manga characters." The server verifies the user's ID and password, confirms access rights, provides an access link, and logs the access time and content.

[0506] input:

[0507] User login information (ID, password)

[0508] Content Request

[0509] Data processing:

[0510] Verification of user authentication information

[0511] Access rights verification

[0512] Access log recording

[0513] output:

[0514] Access links to content for which access rights have been verified.

[0515] Recorded access logs

[0516] (Application Example 1)

[0517] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0518] Traditionally, systems that provide character data as virtual reality (VR) or augmented reality (AR) content have struggled to efficiently integrate multiple functions such as data digitization, content generation, content distribution, and user access management. Furthermore, there was a lack of technology to effectively display characters in the user's real-world environment and provide an interactive experience. As a result, it was difficult for users to obtain a consistent AR experience.

[0519] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0520] In this invention, the server includes means for generating character data converted into a digital format, means for generating content for virtual reality or augmented reality based on the character information converted into a digital format, means for distributing the generated content to a corresponding platform, means for verifying access rights to content requested by the user and recording access logs, means for acquiring a camera feed from a smart device using augmented reality software and displaying a 3D model on the acquired camera feed, and means for playing audio that interacts with the 3D model. This enables the user to naturally display the character in the real environment and seamlessly enjoy an interactive experience.

[0521] A "digital format" refers to a form that can be stored and handled as digital data.

[0522] "Character data" refers to information about characters that appear in manga, anime, games, etc., including images, models, voices, and movements.

[0523] "Virtual reality (VR)" refers to a technology that provides interactive experiences within a virtual 3D space created using computer technology.

[0524] Augmented reality (AR) refers to a technology that overlays virtual information onto the real, physical environment.

[0525] "Content" refers to information and media (such as videos, audio, and 3D models) that users consume.

[0526] A "platform" refers to an environment, infrastructure, or service that provides the foundation for software and hardware to operate.

[0527] "Access rights" refer to permission to operate on or use specific content or services.

[0528] "Access logs" refer to data that records the history of how users have accessed a system or content.

[0529] "Augmented reality software" refers to software that overlays digital information onto the real world.

[0530] "Smart devices" refer to mobile terminals, eyeglasses, and wearable devices that have internet connectivity and high processing power.

[0531] "Camera feed" refers to real-time video footage captured by a camera.

[0532] A "3D model" refers to digital data that simulates a three-dimensional object, created using a computer.

[0533] "Audio playback means" refers to devices or functions for playing audio data through devices such as speakers or headphones.

[0534] "Interaction" refers to the two-way exchange that takes place between a user and a digital system or content.

[0535] The system realizing this invention has the function of collecting and generating character data converted into a digital format, generating virtual reality (VR) or augmented reality (AR) content based on that data, and distributing it to users. The specific configuration and operation of this system are described below.

[0536] 1. Character data collection and digitization

[0537] The server first receives character data from content providers (such as publishing companies or game companies). This data includes information such as character images, 3D models, audio clips, and motion data. This character data is then converted into a digital format using scanning and modeling technologies. This conversion enables the creation of high-quality digital content.

[0538] 2. Generating VR / AR content

[0539] The server generates VR / AR content based on character data converted into a digital format. Here, for example, 3D modeling and motion capture technology are used to create interactive characters with natural movements. Voice recognition functionality is also integrated to create characters that respond to user voices.

[0540] 3. Content distribution

[0541] The server distributes the generated VR / AR content to the corresponding platform. This platform includes dedicated applications, online platforms, and game consoles. By using smart devices (e.g., smartphones and smart glasses), users can have an interactive experience.

[0542] 4. Realization of Augmented Reality

[0543] The server uses augmented reality software to retrieve the camera feed from the smart device. A 3D model of the character is then overlaid onto the retrieved camera feed. For example, if the user is wearing smart glasses, the character will appear in the real-world scene, allowing for interaction with its movements and sounds.

[0544] 5. User Access Management

[0545] The server checks the user's access rights to the requested content and logs the access if it is granted. This log is used for later analysis and service improvement.

[0546] Explanation of specific examples

[0547] For example, suppose a user wants to enjoy an AR experience with a famous character. When the user opens a dedicated application and puts on smart glasses, the server delivers a 3D model and voice data of the character. As the user walks around in the real world, the character appears in that location and speaks and acts in response to the user's voice. Such an experience makes it feel as if the character is actually present in reality.

[0548] Example of a prompt

[0549] Design an application that provides an AR experience using a 3D model and voice data of Naruto. The application will allow users to interact with the character in real time using smart glasses. The character's movements and voice should be natural and interactive.

[0550] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0551] Step 1:

[0552] The server retrieves character data (images, 3D models, audio clips, motion data, etc.) from content providers. This data is accessed directly from the content provider's database or received as a provided file. The retrieved data is temporarily stored on the server. The input is character data, and the output is raw character data stored on the server.

[0553] Step 2:

[0554] The server converts the acquired character data into a digital format. For example, it can digitize hand-drawn illustrations using scanning technology or convert existing 2D images into 3D models. This conversion process uses specialized software (e.g., Adobe Photoshop, Blender, etc.). The input is raw character data, and the output is character data converted into a digital format.

[0555] Step 3:

[0556] The server generates VR / AR content based on character data converted into a digital format. This process uses modeling software to recreate natural character movements and incorporates voice recognition and interactive features. For example, motion capture technology is used to control character movements in real time. The input is character data in a digital format, and the output is content optimized for VR / AR.

[0557] Step 4:

[0558] The server distributes the generated VR / AR content to the corresponding platform. This platform includes dedicated applications, online platforms, and apps for smart devices, and the content needs to be converted to an appropriate format for each platform. The input is the VR / AR content, and the output is the content converted to a format suitable for the platform.

[0559] Step 5:

[0560] The device acquires the camera feed from a smart device. A 3D character model received from a server is overlaid on the acquired real-time video data. Specifically, augmented reality software is used to position the character according to real-world coordinates and enable interactive actions. The input is the camera feed and the 3D character model, and the output is the video displayed in augmented reality.

[0561] Step 6:

[0562] The device receives voice input from the user and converts it to text using a speech recognition engine (e.g., Google Speech-to-Text API). This text is then analyzed to determine how the character should react. The input is the user's voice, and the output is the text data and the character's reaction.

[0563] Step 7:

[0564] The server manages user access rights and verifies access rights to requested content. If access is granted, it logs that information and grants the user access to the content. The input is the user's request information, and the output is log data regarding access rights and the access permissions.

[0565] Step 8:

[0566] The server analyzes user access logs to understand popular content and user behavior patterns. This analysis is used to improve services and develop new content. The input is access log data, and the output is the analysis results.

[0567] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0568] This invention is a system that converts character data into a digital format, generates virtual reality (VR) or augmented reality (AR) content, and delivers that content to a platform. Furthermore, it incorporates an emotion engine that recognizes user emotions and dynamically changes the content accordingly.

[0569] 1. Character data collection and digitization

[0570] The server first collects character data from content providers (such as publishing companies and game companies). This data includes images, voices, text, and 3D models of each character. The collected data is then converted into a digital format. This process utilizes high-resolution image scanning and 3D modeling software.

[0571] 2. Generating VR / AR content

[0572] The server generates VR and AR content based on digitized character data. This content can include 3D models, audio, and movement. It also designs specific algorithms and scenarios to allow users to interact with the characters.

[0573] 3. Integrating an emotion engine

[0574] The server incorporates an emotion engine that recognizes the user's emotions in real time. The emotion engine uses cameras and microphones to analyze the user's facial expressions, voice, and gestures to determine the user's emotional state. For example, if the user is smiling, it determines they are "happy," and if they are frowning, it determines they are "sad."

[0575] 4. Dynamic content changes based on emotions

[0576] The server dynamically changes the VR / AR content based on the user's emotions, as recognized by the emotion engine. For example, if the user is perceived as "happy," the character will perform more friendly actions. Conversely, if the user is perceived as "sad," the character will perform comforting actions.

[0577] 5. Content distribution

[0578] The server distributes the generated VR / AR content to the appropriate platform (such as a dedicated application, online platform, or game console). It delivers the content in a format suitable for each platform, ensuring easy access for users.

[0579] 6. User Access Management

[0580] The server verifies the user's access rights to the requested content. Users whose access rights are verified are granted access to the content, and an access log is recorded. This log includes the user ID, access date and time, and the type of content accessed.

[0581] Specific example

[0582] 1. Specific examples of character data collection and digitization

[0583] The server receives data for "Popular Character A" from the content provider. This data includes a full-body image, audio clips, and character design documents. The server then converts this data into a digital format and creates a high-resolution 3D model.

[0584] 2. Specific examples of VR / AR content generation

[0585] The server generates VR content for "Popular Character A" based on the created 3D model. Motion capture data is used to give the character realistic movements, and voice recognition functionality is added to implement user interaction.

[0586] 3. Specific Examples of Emotional Engines

[0587] The server activates an emotion engine to analyze the user's facial expressions and voice. For example, if the user is smiling, it recognizes that they are "happy," and if they are frowning, it recognizes that they are "sad."

[0588] 4. Specific examples of dynamic content changes based on emotions

[0589] The server configures the VR content's characters to perform friendly actions and speak in a way that indicates the user is "happy." Conversely, if the user is perceived as "sad," the characters' actions are adjusted to offer comforting words.

[0590] 5. Specific examples of content distribution

[0591] The server uploads the generated VR content of "Popular Character A" to a dedicated application. Users can access this content through the dedicated application and enjoy the experience on their home VR headset.

[0592] 6. Specific Examples of User Access Management

[0593] A user attempts to access VR content featuring "Popular Character A." The server verifies the user's authentication information and confirms access rights. If access is permitted, the user's access log is recorded, and their usage history for the content is managed.

[0594] This enables the system of the present invention to efficiently generate advanced character content and provide interactive experiences that respond to user emotions. This system allows character businesses in the Japanese entertainment sector to effectively expand on a new platform, providing compelling experiences to users worldwide.

[0595] The following describes the processing flow.

[0596] Step 1:

[0597] The server collects character data from content providers. This data includes character images, audio, text, and 3D models. For example, it might receive full-color illustrations of manga characters or audio data of character dialogue.

[0598] Step 2:

[0599] The server converts the received character data into a digital format. This conversion includes scanning the image data into high-resolution electronic files and generating a 3D model from the scanned data. For example, software is used to create 3D models from 2D illustrations.

[0600] Step 3:

[0601] The server generates VR and AR content based on character data converted into a digital format. Specifically, it adds movement to 3D models and implements speech recognition and generation functions to enable dialogue. Motion capture technology is used to capture natural character movements.

[0602] Step 4:

[0603] The server activates an emotion engine to analyze the user's facial expressions and voice in real time. It uses cameras and microphones to recognize the user's emotions, and the emotion engine analyzes the acquired data. For example, if the user is smiling, it determines they are "happy," and if they are frowning, it recognizes them as "sad."

[0604] Step 5:

[0605] The server dynamically modifies VR and AR content based on the recognized user's emotions. The character's actions and dialogue change according to the emotion engine's results. For example, if the user is judged to be "happy," the character will respond with a smile. If the user is judged to be "sad," the character will perform comforting actions or speak comforting words.

[0606] Step 6:

[0607] The server distributes the generated VR / AR content to the appropriate platform. It distributes the content to dedicated applications, online platforms, game consoles, etc., making it accessible to users. For example, it uploads VR content to a dedicated app, allowing users to download it.

[0608] Step 7:

[0609] A user attempts to access VR / AR content. The server checks the user's access rights and authenticates whether they have the appropriate permissions. If the user has the necessary permissions, access to the content is permitted.

[0610] Step 8:

[0611] The server records user access logs. It records which content users accessed and when, and uses this information for later analysis and operational management. For example, it stores user IDs, access dates and times, and the type of content accessed in a database.

[0612] Step 9:

[0613] Users can access permitted content. They can enjoy interactive experiences using VR headsets and AR-enabled devices. For example, they can wear a VR headset and interact with characters in a virtual space. They can also use AR devices to make characters appear in the real world.

[0614] The above outlines the specific processing flow in the system of the present invention that incorporates an emotion engine. This step-by-step processing makes it possible to manage everything from character data collection to user experience efficiently and interactively.

[0615] (Example 2)

[0616] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0617] In recent years, virtual reality (VR) and augmented reality (AR) technologies have developed rapidly, and many users are seeking interactive and engaging content. However, current systems struggle to recognize user emotions in real time and dynamically change content accordingly. As a result, the quality of the user experience is limited, and in some scenarios, there is a lack of immersion. Furthermore, the effort and resources required to support multiple platforms also remain challenges.

[0618] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0619] In this invention, the server includes means for generating character data converted into a digital format, means for generating content for virtual reality or augmented reality based on the character information converted into a digital format, means for distributing the generated content to a corresponding platform, means for verifying access rights to content requested by the user and recording access logs, an emotion recognition engine that recognizes the user's emotions in real time using a camera and microphone, and means for dynamically changing the content for virtual reality or augmented reality based on the user's emotions. This enables real-time content adjustment in response to the user's emotions, resulting in a more immersive and interactive experience.

[0620] A "digital format" is a format in which physical analog data is converted into digital data that can be used by a computer.

[0621] "Character data" refers to information about characters used in virtual reality and augmented reality content, and includes images, audio, text, 3D models, and more.

[0622] Virtual reality (VR) is a technology that allows humans to experience virtual spaces and objects created using computer technology through their five senses.

[0623] Augmented reality (AR) is a technology that overlays digital information onto real-world scenery, integrating computer-generated visual elements into the real world for display.

[0624] A "content provider" refers to a company or organization that generates and distributes digital content, and this includes publishing companies, game companies, and others.

[0625] A "platform" refers to the underlying environment or services for running and providing software and content, and includes dedicated applications, online platforms, and game consoles.

[0626] "Access rights" refer to the right of a user to access specific digital content or resources.

[0627] An "access log" is data that records the history of access to a system or database, and includes information such as the user's ID, the date and time of access, and the type of content accessed.

[0628] An "emotion recognition engine" is a software engine that uses sensors such as cameras and microphones to analyze a user's facial expressions, voice, and gestures, and recognizes their emotional state in real time.

[0629] "Dynamic modification" refers to changing the operation or output of a system in real time or on an ad-hoc basis in response to specific conditions or data.

[0630] This invention is a system that converts character data into a digital format, generates virtual reality (VR) or augmented reality (AR) content, and delivers that content to a platform. Furthermore, it incorporates an emotion recognition engine that recognizes the user's emotions and dynamically changes the content accordingly.

[0631] Character data collection and digitization

[0632] The server first collects character data from content providers (such as publishing companies and game companies). The collected data includes images, voices, text, and 3D models for each character. Next, this data is converted into digital formats using high-resolution image scanning and 3D modeling software. Specifically, image data is edited using Adobe Photoshop, converted to high-resolution images, and 3D models are generated using Autodesk Maya.

[0633] VR / AR content generation

[0634] The server generates VR and AR content based on digitized character data. This content includes 3D models, audio, and motion. Dialogue scenarios and interaction algorithms are designed to allow users to interact with the characters. Specific development utilizes game engines such as Unity and Unreal Engine, imports motion capture data, and uses the Google Cloud Speech-to-Text API for speech recognition.

[0635] Embedding an emotion recognition engine

[0636] The server incorporates an emotion recognition engine that uses cameras and microphones to recognize the user's emotions in real time. This engine analyzes the user's facial expressions using face recognition libraries such as OpenCV and Dlib, and recognizes emotions from audio data using Microsoft Azure's Emotion API.

[0637] As a concrete example, when a user connects to the system, the camera captures the user's face and analyzes their facial expressions using OpenCV. Audio data from the microphone is processed through the Emotion API, and the user's emotions are determined in real time.

[0638] Dynamic content changes based on emotions

[0639] The server dynamically modifies VR / AR content based on the user's emotions recognized by the emotion recognition engine. For example, if the user is perceived as "happy," the character will perform more friendly actions, and if the user is perceived as "sad," comforting actions will be performed. This process is carried out using an action script (e.g., Python), which calls appropriate animations and dialogue based on the specific emotional state.

[0640] Content distribution

[0641] The server delivers the generated VR / AR content to the appropriate platform. It delivers the content in a format compatible with dedicated applications, online platforms, game consoles, etc., ensuring easy access for users. Specifically, it uses a system that uploads content to an AWS S3 bucket and delivers it to users worldwide with low latency using a CDN (Content Delivery Network).

[0642] User access management

[0643] The server verifies the user's access rights to the requested content. Users who are granted access are allowed to access the content, and an access log is recorded. This log includes information such as the user ID, access date and time, and the type of content accessed. Authentication is performed using OAuth or JWT (JSON Web Token), and the access log is recorded in a database such as MySQL or MongoDB.

[0644] Specific example

[0645] 1. Specific examples of character data collection and digitization

[0646] The server downloads full-body images, audio clips, and setting materials of "Popular Character A" via the publishing company's API, digitizes the image data using Adobe Photoshop, and creates a high-resolution 3D model using Autodesk Maya.

[0647] 2. Specific examples of VR / AR content generation

[0648] The server uses the Unity engine to add movement to a 3D model of "Popular Character A," implements speech recognition functionality using the Google Cloud Speech-to-Text API, and designs user interaction scenarios.

[0649] 3. Specific Examples of Emotion Recognition Engines

[0650] The server uses OpenCV to capture the user's facial expressions in real time and the Microsoft Azure Emotion API to recognize emotions from the audio data. It determines the emotional state, such as "happy" if the user is smiling and "sad" if they are frowning.

[0651] 4. Specific examples of dynamic content changes based on emotions

[0652] If the server determines that the user is "happy," the character will smile and speak in a friendly manner; if it determines that the user is "sad," the character will offer words of comfort.

[0653] 5. Specific examples of content distribution

[0654] The server uploads VR content of "Popular Character A" generated in an AWS S3 bucket, and users can access the content through a dedicated application and experience it using their home VR headset.

[0655] 6. Specific Examples of User Access Management

[0656] The user logs into the app to access VR content featuring "Popular Character A," and the server performs user authentication using OAuth. If authentication is successful, the user's access log is recorded in a MySQL database.

[0657] This system enables real-time content adjustments based on user emotions, resulting in a more immersive and interactive experience.

[0658] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0659] Step 1: Collecting character data

[0660] The server collects character data from content providers. The input is character data (images, audio, text, 3D models) provided by publishing companies and game companies. The output is stored on the server as collected data. Specifically, the server sends API requests to content providers to retrieve the data and stores it in its own storage.

[0661] Step 2: Digitizing character data

[0662] The server converts the collected character data into a digital format. The input is the collected analog data or low-resolution digital data. The output is character data formatted as high-resolution digital data. Specifically, Adobe Photoshop is used to convert the image data to high resolution, and Autodesk Maya is used to create a 3D model.

[0663] Step 3: Generating VR / AR content

[0664] The server generates VR and AR content based on digitized character data. Inputs include digitized 3D models and audio data. Output is interactive VR / AR content. For specific actions, Unity or Unreal Engine is used to add movement to the 3D models, and the Google Cloud Speech-to-Text API is used to implement speech recognition functionality.

[0665] Step 4: Start the emotion recognition engine

[0666] The server activates an emotion recognition engine that recognizes the user's emotions in real time. The input consists of facial video and audio data sent from the user's terminal. The output is the emotion recognition result. Specifically, the server uses OpenCV and Dlib to analyze video captured from the camera, and the Microsoft Azure Emotion API to analyze audio data.

[0667] Step 5: Dynamically change content based on emotions

[0668] The server dynamically modifies VR / AR content based on the user's emotions recognized by the emotion recognition engine. The input is the emotion recognition result. The output is VR / AR content optimized according to the user's emotions. Specifically, the server uses an action script written in Python to call appropriate animations and dialogue in real time based on a specific emotional state.

[0669] Step 6: Content Distribution

[0670] The server delivers the generated VR / AR content to the corresponding platform. The input is the completed VR / AR content. The output is the content provided in a state accessible to the user. Specifically, the content is uploaded to an AWS S3 bucket and delivered to users worldwide with low latency using a CDN.

[0671] Step 7: Managing User Access

[0672] The server verifies the user's access rights to the requested content and records access logs. The input is the user's authentication information. The output is the content the user can access after authentication, and log data representing the access history. Specifically, authentication is performed using OAuth or JWT, and the results are recorded in a MySQL or MongoDB database.

[0673] This processing flow allows for the rapid and efficient delivery of interactive VR / AR experiences that respond to the user's emotions.

[0674] (Application Example 2)

[0675] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0676] Traditional virtual reality (VR) and augmented reality (AR) content systems face the challenge of providing interactive experiences that respond to user emotional changes. Furthermore, users are often limited to certain patterns of actions and scenarios when enjoying the content, resulting in a lack of realism and immersion. Additionally, there is a lack of technology to dynamically modify content based on real-time user feedback.

[0677] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for generating character data converted into a digital format, means for generating content for virtual reality or augmented reality based on the character information converted into a digital format, means for distributing the generated content to a corresponding platform, means for confirming access rights to the content requested by the user and recording access logs, means for analyzing the user's facial expressions and voice to recognize emotions, and means for dynamically changing the content based on the recognized emotions. This makes it possible to provide an interactive VR / AR experience that responds to the user's emotional state.

[0678] A "digital format" is a format in which analog data has been converted into a format that can be processed on a computer.

[0679] "Character data" refers to information such as images, audio, 3D models, and text related to a specific character.

[0680] "Virtual reality" is a technology that allows users to experience a three-dimensional artificial environment created using computer technology, as if it were reality, through their sight and hearing.

[0681] Augmented reality is a technology that overlays virtual information onto the real world, allowing users to view both the real world and digital information simultaneously.

[0682] A "platform" refers to the underlying software and hardware environment on which applications and content operate.

[0683] "Access rights" refer to the right of a user to access specific content or information, and authentication determines whether a user is entitled to access that information.

[0684] An "access log" is a record of a user's access history kept by the system, and includes information such as the user ID, the date and time of access, and the type of content accessed.

[0685] "Facial expression" refers to emotions and reactions that are expressed through the movement of the facial structure.

[0686] "Sound" refers to sounds that are transmitted as sound waves and can be heard by the human ear, and mainly includes human speech.

[0687] "Emotion" refers to the psychological response or state that a user experiences in response to a particular situation or stimulus.

[0688] "Dynamic modification" means instantly changing content and information in response to real-time changes in circumstances.

[0689] "Interactive" refers to a system that interacts with the user and responds immediately to user input.

[0690] This invention relates to a system that converts character data into a digital format, generates virtual reality (VR) or augmented reality (AR) content, and incorporates an emotion engine that recognizes user emotions and dynamically changes the content. This system is implemented using the following steps, hardware, and software.

[0691] 1. Collection and digitization of character data

[0692] The server first collects character data from content providers. This data includes images, audio, text, and 3D models of each character. This collected data is then converted into a digital format and stored. This process utilizes high-resolution image scanning and 3D modeling software.

[0693] 2. Generating VR / AR content

[0694] The server generates VR and AR content based on character data converted into a digital format. At this stage, elements such as 3D models, voice, and movement are combined. Furthermore, certain algorithms and scenarios are designed to enable interactive dialogue.

[0695] 3. Integrating an emotion engine

[0696] An emotion engine has been introduced to recognize the user's emotions in real time. Using cameras and microphones, it collects data on the user's facial expressions, voice, and gestures, and uses this data to determine their emotional state. For example, if the user is smiling, it will be determined to be "happy," and if they are frowning, it will be determined to be "sad."

[0697] 4. Dynamic content changes based on emotions

[0698] The server dynamically changes the VR / AR content based on the user's emotions, as recognized by the emotion engine. For example, if the user is perceived as "happy," the character will perform more friendly actions. Conversely, if the user is perceived as "sad," the character will perform comforting actions.

[0699] 5. Content distribution

[0700] The generated VR / AR content is delivered to the appropriate platform (such as a dedicated application, online platform, or game console). The content is delivered in a format suitable for each platform, ensuring easy access for users.

[0701] 6. User Access Management

[0702] The server checks the user's access rights to the requested content and records access logs as needed. The access logs include the user ID, access date and time, and the type of content accessed.

[0703] Hardware and software to use

[0704] This system uses the following hardware and software.

[0705] Hardware: Webcam, VR headset, smartphone

[0706] Software: Python, OpenCV, Dlib, Pyttsx3, 3D modeling software

[0707] Specific example

[0708] For example, in a system applying the present invention, suppose a user puts on a VR headset and begins interacting with "Popular Character A". If the user smiles, the animation changes to the character speaking in a friendly manner. Conversely, if the user shows a sad expression, the scenario changes to one in which the character offers words of comfort.

[0709] Example of a prompt

[0710] Please create a VR character that responds in a friendly manner when the user is smiling and in a comforting manner when the user is sad.

[0711] The necessary data is to read the user's emotions from their facial expressions and change the behavior based on that.

[0712] Use Python's OpenCV and Dlib libraries to perform emotion recognition, and the Python library pyttsx3 to provide voice feedback.

[0713] Thus, this invention provides an interactive VR / AR experience based on the user's emotions, enabling a more personalized and engaging experience for the user.

[0714] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0715] Step 1:

[0716] This system collects and digitizes character data. The server collects data such as images, audio, text, and 3D models related to characters from content providers. Input is character data, and output is character data converted to a digital format. High-resolution image scanning and 3D modeling software are used to convert the data to a digital format and save it.

[0717] Step 2:

[0718] This system generates VR / AR content. The server uses character data converted to a digital format to create VR and AR content by combining 3D models, audio, and movement. Input is character data in digital format, and output is VR / AR content. Specific algorithms and scenarios are also designed to enable interactive dialogue.

[0719] Step 3:

[0720] An emotion engine will be implemented. The server will use the emotion engine to analyze the user's facial expressions and voice to recognize their emotions. The input will be user facial expression and voice data collected by cameras and microphones, and the output will be the recognized emotional state. Data will be collected using cameras and microphones, and the user's emotions will be determined using software such as OpenCV or Dlib.

[0721] Step 4:

[0722] The content is dynamically changed according to emotions. The server dynamically changes the VR / AR content based on the user's emotions recognized by the emotion engine. The input is the recognized emotional state, and the output is the dynamically changed VR / AR content. For example, if the user is recognized as "happy," the character is set to perform friendly actions.

[0723] Step 5:

[0724] The content is delivered to the platform. The server uploads the generated VR / AR content to a dedicated application or online platform. The input is the generated VR / AR content, and the output is the content accessible on the platform. The content is delivered in an appropriate format, making it easily accessible to users.

[0725] Step 6:

[0726] This system manages user access. The server verifies user access rights to requested content and records access logs. Inputs include user requests and authentication information, while outputs include access rights verification and access logs. Users whose access rights have been verified are granted access to the content, and the system records user ID, access date and time, and the type of content accessed.

[0727] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0728] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0729] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0730] [Third Embodiment]

[0731] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0732] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0733] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0734] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0735] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0736] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0737] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0738] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0739] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0740] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0741] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0742] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0743] The system for carrying out the present invention will now be described. This system consists of the following main components.

[0744] 1. Character data collection and digitization

[0745] The server first receives character data from content providers (e.g., manga and anime publishers, game companies, etc.). This includes information such as character images, models, voices, and movements. The received character data is then converted into a digital format. This conversion process uses technologies such as scanning to digitize hand-drawn illustrations or digitizing 3D models.

[0746] 2. Generating VR / AR content

[0747] The server generates VR / AR content based on character information converted into a digital format. For example, it 3D models a manga character and adds movement and sound. In this generation process, algorithms for controlling character movements in real time and interactive scenarios to respond to user movements are designed.

[0748] 3. Content distribution

[0749] The server distributes the generated VR / AR content to the corresponding platform. This platform includes dedicated applications, online platforms, and game consoles. The server distributes the content in a format suitable for each platform and adjusts it to ensure smooth access for users.

[0750] 4. User Access Management

[0751] The server verifies the user's access rights to the requested content. It checks whether the user has the appropriate permissions and grants access to the content if they do. It records logs of when access is granted and uses them for later analysis and operation. This makes it possible to understand user behavior history and popular content.

[0752] Specific example

[0753] Specific examples are given below.

[0754] 1. Specific examples of character data collection and digitization

[0755] The server receives data of a "famous cartoon character" from a content provider. This data includes a full-body image of the character, audio clips, and character design documents. The server then converts this data into a digital format and creates a 3D model.

[0756] 2. Specific examples of VR / AR content generation

[0757] The server generates VR content of "famous manga characters" based on the created 3D models. Motion capture data is used to make the characters' movements appear natural. Furthermore, voice recognition functionality is incorporated to allow users to interact with the characters.

[0758] 3. Specific examples of content distribution

[0759] The server uploads the generated VR content of "famous cartoon characters" to a dedicated application. Users can download this content through the application and enjoy it on their VR headsets at home.

[0760] 4. Specific Examples of User Access Management

[0761] Users attempt to access VR content featuring "famous manga characters" through a dedicated application. The server verifies whether the user has access rights and, if access is permitted, records the user's access log. This ensures that users can enjoy the content safely and smoothly.

[0762] This invention realizes a system that efficiently and effectively handles everything from the digitization of character data to the generation and distribution of VR / AR content, as well as user access management. This will enable the development of character businesses in the Japanese entertainment sector on a new platform and make them available to users worldwide.

[0763] The following describes the processing flow.

[0764] Step 1:

[0765] The server collects character data from content providers. This character data includes a wide variety of information, such as images, audio, text, and 3D models. For example, various illustrations of manga characters and animation direction materials are provided.

[0766] Step 2:

[0767] The server converts the collected character data into a digital format. For example, it might scan hand-drawn illustrations into high-resolution image files or create character models using 3D modeling software.

[0768] Step 3:

[0769] The server generates VR (virtual reality) or AR (augmented reality) content based on character data converted into a digital format. For example, it might add movement to 3D models or implement voice recognition functionality. This creates content that allows users to interact and communicate with the characters.

[0770] Step 4:

[0771] The server distributes the generated VR / AR content to the specified platform. This distribution destination can be a dedicated application, an online platform, or a game console. For example, it might package the content into a compressed file for a dedicated application and upload it.

[0772] Step 5:

[0773] The server verifies the user's access rights. When a user attempts to access content, the server checks whether they have the appropriate permissions based on their authentication information. For example, it checks whether they are a paid member or have purchased a specific license.

[0774] Step 6:

[0775] If a user has the appropriate access rights, the server will grant them access to the content. The server records logs of users who have been granted access for later analysis and operational management. For example, it records user ID, access date and time, and the type of content accessed.

[0776] Step 7:

[0777] Users can access permitted content. They can enjoy interactive experiences using VR headsets and AR-enabled devices. For example, they can wear a VR headset to interact with cartoon characters in a virtual space, or use AR devices to make characters appear in the real world.

[0778] The above outlines the specific processing flow in the system of the present invention. This step-by-step processing makes it possible to efficiently manage everything from character data collection to user experience.

[0779] (Example 1)

[0780] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0781] In systems that generate virtual reality (VR) or augmented reality (AR) content based on existing character data, allowing users to access and experience it in real time, there is a need to streamline the content generation process and manage user access more securely and effectively. In particular, there is a lack of systems that can smoothly digitize hand-drawn illustrations and 3D models and design interactive scenarios based on them.

[0782] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0783] In this invention, the server includes means for acquiring content data from a content provider and converting it into a digital format; means for generating content for virtual reality or augmented reality based on the digitally converted content data; means for designing interactive scenarios for the generated content so that users can interact with the content in real time; means for delivering the generated virtual reality or augmented reality content to a corresponding platform; and means for verifying user access rights to the requested content and recording access logs. This makes it possible to consistently and efficiently carry out everything from content generation to delivery and user access management.

[0784] A "content provider" is an organization or individual that possesses character data to be converted into a digital format and provides that data to the server.

[0785] "Content data" refers to information necessary for virtual reality and augmented reality content, such as character images, voices, 3D models, and movements.

[0786] A "digital format" refers to a format of digital data that can be processed by a computer.

[0787] "Virtual reality" is a technology that immerses users in a computer-generated three-dimensional environment.

[0788] Augmented reality is a technology that overlays digital information onto images of the real world.

[0789] An "interactive scenario" refers to the design of a story or event that changes dynamically in response to user input and actions.

[0790] A "platform" refers to an application or online system used to deliver virtual reality or augmented reality content.

[0791] "Access rights" refer to the permissions a user has to access specific content.

[0792] An "access log" refers to a record of when a user accesses a system, and includes information such as the date and time and the content accessed.

[0793] "Hand-drawn illustrations" refer to images drawn by hand on paper or other physical media, and include converting them into a digital format.

[0794] A "3D model" refers to a three-dimensional character or object generated by a computer.

[0795] A "dedicated application" refers to specific software intended for the playback or use of virtual reality or augmented reality content.

[0796] An "online platform" refers to a website or cloud system that provides virtual reality or augmented reality content via the internet.

[0797] A "dedicated terminal" refers to a specific hardware device used to experience virtual reality or augmented reality content.

[0798] This invention relates to a system that generates virtual reality (VR) or augmented reality (AR) content based on character data, which users can access in real time. This system consists of three main elements: a server, a terminal, and a user.

[0799] Character data collection and digitization

[0800] The server receives character data from content providers. This character data includes information such as character images, 3D models, voice, and movements. The server converts the received data into a digital format. Specifically, this involves using technologies to digitize hand-drawn illustrations using high-resolution scanners, or technologies to generate 3D models using 3D scanners.

[0801] For example, we can digitize full-body images, audio clips, and character design documents of a "famous manga character" and convert them into 3D models.

[0802] VR / AR content generation

[0803] The server generates VR / AR content based on digitized character information. Specifically, it uses 3D modeling software (e.g., Blender, Maya) to fine-tune the character's 3D model and imports motion capture data to add natural movements. Furthermore, it uses speech recognition software (e.g., Google Speech-to-Text) to generate the character's voice. The server then uses the Unity engine to create a VR / AR scene using the generated 3D model and voice data.

[0804] For example, you can optimize the polygon count of a character's 3D model in Blender and then use the Unity engine to create a scene where the character walks and talks.

[0805] Content distribution

[0806] The server distributes the generated VR / AR content to the corresponding platform. Specifically, it uploads the content to a dedicated application (e.g., Android app, iOS app) and uses an API to deliver the content to the user's device in the appropriate format. If the content is updated, the server automatically distributes the new version.

[0807] For example, VR content of a "famous manga character" that has been generated can be uploaded to a dedicated application, allowing users to download this content through the app.

[0808] User access management

[0809] The server verifies the user's access rights to the requested content and records access logs. Specifically, when a user logs into a dedicated application and submits a request for content, the server compares the user's authentication information with the database to check for access rights. If access is permitted, the server provides the user with an access link to the content and records the access log.

[0810] As a concrete example, when a user attempts to access VR content featuring a "famous manga character" using a dedicated application, the server verifies the user's login information, provides an access link if they have access rights, and records an access log.

[0811] Specific examples and prompt statements

[0812] Example 1: The server downloads character data from "http: / / provider.example.com / character1.zip", unzips the ZIP file, converts it to PNG format, and saves it to the database.

[0813] Example 2: The server uses Blender to optimize a 3D model of a "famous anime character" and then creates a VR scene by adding motion capture data in Unity.

[0814] Specific example 3: The server uploads the generated VR content of "famous manga characters" to a dedicated application, and the user downloads it through the application.

[0815] Specific example 4: When a user attempts to access VR content of a "famous manga character" using a dedicated application, the server verifies the user's login information, provides an access link if they have access rights, and records an access log.

[0816] Example of a prompt:

[0817] "Generate VR / AR content based on data of famous manga characters, verify user access rights, and distribute the content."

[0818] This system enables efficient and effective digitization of character data, generation and distribution of VR / AR content, and management of user access. This allows for the development of character-based businesses in the entertainment sector on a new platform, providing engaging content to users worldwide.

[0819] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0820] Step 1:

[0821] Character data collection and digitization

[0822] The server receives character data from content providers. This character data includes image files, audio files, 3D models, etc. The server converts this data into a digital format and stores it in a dedicated database.

[0823] Specifically, the server downloads character data from a specified URL. For example, it downloads a ZIP file from "http: / / provider.example.com / character1.zip", extracts PNG image files and FBX 3D models, and then stores this data in a database.

[0824] input:

[0825] Character data (images, audio, 3D models) from content providers

[0826] Data processing:

[0827] Downloading and extracting ZIP files

[0828] Convert image files to PNG format and 3D models to FBX format.

[0829] output:

[0830] Character data converted to digital format (PNG image files, FBX 3D models, etc.)

[0831] Step 2:

[0832] VR / AR content generation

[0833] The server generates VR / AR content based on digitized character information. Specifically, it uses 3D modeling software (Blender, Maya) to fine-tune the character's 3D model and imports motion capture data to add natural movements to the character. Furthermore, it uses speech recognition software (Google Speech-to-Text) to generate the character's voice. Finally, the server uses the Unity engine to create a VR / AR scene using the generated 3D model and voice data.

[0834] Specifically, the process involves using Blender to optimize the polygon count of the character's 3D model and then using the Unity engine to create scenes where the character walks and talks.

[0835] input:

[0836] Digitized character data (PNG image files, FBX 3D models, etc.)

[0837] Data processing:

[0838] Detailed adjustments in 3D modeling software

[0839] Importing motion capture data

[0840] Generate audio data using speech recognition software

[0841] Creating VR / AR scenes with the Unity engine

[0842] output:

[0843] VR / AR content (scenes integrating 3D models, motion, and sound)

[0844] Step 3:

[0845] Content distribution

[0846] The server distributes the generated VR / AR content to the corresponding platform. Specifically, it uploads the content to a dedicated application (Android app, iOS app) and uses an API to deliver the content to the user's device in the appropriate format. If the content is updated, the server automatically distributes the new version.

[0847] Specifically, the generated VR content is uploaded to "https: / / app.example.com / updates," a dedicated application checks for the latest version of the content, and a download notification is displayed to the user.

[0848] input:

[0849] Generated VR / AR content (a scene integrating 3D models, motion, and sound)

[0850] Data processing:

[0851] Upload content

[0852] Delivery adjustment using API

[0853] output:

[0854] Content delivered to the user's device

[0855] Step 4:

[0856] User access management

[0857] The server verifies the user's access rights to the requested content and records access logs. Specifically, when a user logs into a dedicated application and submits a request for content, the server compares the user's authentication information with a database to check for access rights. If access rights are granted, the server provides an access link to the content and records access logs.

[0858] Specifically, the user logs in to a dedicated application to access VR content featuring "famous manga characters." The server verifies the user's ID and password, confirms access rights, provides an access link, and logs the access time and content.

[0859] input:

[0860] User login information (ID, password)

[0861] Content Request

[0862] Data processing:

[0863] Verification of user authentication information

[0864] Access rights verification

[0865] Access log recording

[0866] output:

[0867] Access links to content for which access rights have been verified.

[0868] Recorded access logs

[0869] (Application Example 1)

[0870] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0871] Traditionally, systems that provide character data as virtual reality (VR) or augmented reality (AR) content have struggled to efficiently integrate multiple functions such as data digitization, content generation, content distribution, and user access management. Furthermore, there was a lack of technology to effectively display characters in the user's real-world environment and provide an interactive experience. As a result, it was difficult for users to obtain a consistent AR experience.

[0872] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0873] In this invention, the server includes means for generating character data converted into a digital format, means for generating content for virtual reality or augmented reality based on the character information converted into a digital format, means for distributing the generated content to a corresponding platform, means for verifying access rights to content requested by the user and recording access logs, means for acquiring a camera feed from a smart device using augmented reality software and displaying a 3D model on the acquired camera feed, and means for playing audio that interacts with the 3D model. This enables the user to naturally display the character in the real environment and seamlessly enjoy an interactive experience.

[0874] A "digital format" refers to a form that can be stored and handled as digital data.

[0875] "Character data" refers to information about characters that appear in manga, anime, games, etc., including images, models, voices, and movements.

[0876] "Virtual reality (VR)" refers to a technology that provides interactive experiences within a virtual 3D space created using computer technology.

[0877] Augmented reality (AR) refers to a technology that overlays virtual information onto the real, physical environment.

[0878] "Content" refers to information and media (such as videos, audio, and 3D models) that users consume.

[0879] A "platform" refers to an environment, infrastructure, or service that provides the foundation for software and hardware to operate.

[0880] "Access rights" refer to permission to operate on or use specific content or services.

[0881] "Access logs" refer to data that records the history of how users have accessed a system or content.

[0882] "Augmented reality software" refers to software that overlays digital information onto the real world.

[0883] "Smart devices" refer to mobile terminals, eyeglasses, and wearable devices that have internet connectivity and high processing power.

[0884] "Camera feed" refers to real-time video footage captured by a camera.

[0885] A "3D model" refers to digital data that simulates a three-dimensional object, created using a computer.

[0886] "Audio playback means" refers to devices or functions for playing audio data through devices such as speakers or headphones.

[0887] "Interaction" refers to the two-way exchange that takes place between a user and a digital system or content.

[0888] The system realizing this invention has the function of collecting and generating character data converted into a digital format, generating virtual reality (VR) or augmented reality (AR) content based on that data, and distributing it to users. The specific configuration and operation of this system are described below.

[0889] 1. Character data collection and digitization

[0890] The server first receives character data from content providers (such as publishing companies or game companies). This data includes information such as character images, 3D models, audio clips, and motion data. This character data is then converted into a digital format using scanning and modeling technologies. This conversion enables the creation of high-quality digital content.

[0891] 2. Generating VR / AR content

[0892] The server generates VR / AR content based on character data converted into a digital format. Here, for example, 3D modeling and motion capture technology are used to create interactive characters with natural movements. Voice recognition functionality is also integrated to create characters that respond to user voices.

[0893] 3. Content distribution

[0894] The server distributes the generated VR / AR content to the corresponding platform. This platform includes dedicated applications, online platforms, and game consoles. By using smart devices (e.g., smartphones and smart glasses), users can have an interactive experience.

[0895] 4. Realization of Augmented Reality

[0896] The server uses augmented reality software to retrieve the camera feed from the smart device. A 3D model of the character is then overlaid onto the retrieved camera feed. For example, if the user is wearing smart glasses, the character will appear in the real-world scene, allowing for interaction with its movements and sounds.

[0897] 5. User Access Management

[0898] The server checks the user's access rights to the requested content and logs the access if it is granted. This log is used for later analysis and service improvement.

[0899] Explanation of specific examples

[0900] For example, suppose a user wants to enjoy an AR experience with a famous character. When the user opens a dedicated application and puts on smart glasses, the server delivers a 3D model and voice data of the character. As the user walks around in the real world, the character appears in that location and speaks and acts in response to the user's voice. Such an experience makes it feel as if the character is actually present in reality.

[0901] Example of a prompt

[0902] Design an application that provides an AR experience using a 3D model and voice data of Naruto. The application will allow users to interact with the character in real time using smart glasses. The character's movements and voice should be natural and interactive.

[0903] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0904] Step 1:

[0905] The server retrieves character data (images, 3D models, audio clips, motion data, etc.) from content providers. This data is accessed directly from the content provider's database or received as a provided file. The retrieved data is temporarily stored on the server. The input is character data, and the output is raw character data stored on the server.

[0906] Step 2:

[0907] The server converts the acquired character data into a digital format. For example, it can digitize hand-drawn illustrations using scanning technology or convert existing 2D images into 3D models. This conversion process uses specialized software (e.g., Adobe Photoshop, Blender, etc.). The input is raw character data, and the output is character data converted into a digital format.

[0908] Step 3:

[0909] The server generates VR / AR content based on character data converted into a digital format. This process uses modeling software to recreate natural character movements and incorporates voice recognition and interactive features. For example, motion capture technology is used to control character movements in real time. The input is character data in a digital format, and the output is content optimized for VR / AR.

[0910] Step 4:

[0911] The server distributes the generated VR / AR content to the corresponding platform. This platform includes dedicated applications, online platforms, and apps for smart devices, and the content needs to be converted to an appropriate format for each platform. The input is the VR / AR content, and the output is the content converted to a format suitable for the platform.

[0912] Step 5:

[0913] The device acquires the camera feed from a smart device. A 3D character model received from a server is overlaid on the acquired real-time video data. Specifically, augmented reality software is used to position the character according to real-world coordinates and enable interactive actions. The input is the camera feed and the 3D character model, and the output is the video displayed in augmented reality.

[0914] Step 6:

[0915] The device receives voice input from the user and converts it to text using a speech recognition engine (e.g., Google Speech-to-Text API). This text is then analyzed to determine how the character should react. The input is the user's voice, and the output is the text data and the character's reaction.

[0916] Step 7:

[0917] The server manages user access rights and verifies access rights to requested content. If access is granted, it logs that information and grants the user access to the content. The input is the user's request information, and the output is log data regarding access rights and the access permissions.

[0918] Step 8:

[0919] The server analyzes user access logs to understand popular content and user behavior patterns. This analysis is used to improve services and develop new content. The input is access log data, and the output is the analysis results.

[0920] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0921] This invention is a system that converts character data into a digital format, generates virtual reality (VR) or augmented reality (AR) content, and delivers that content to a platform. Furthermore, it incorporates an emotion engine that recognizes user emotions and dynamically changes the content accordingly.

[0922] 1. Character data collection and digitization

[0923] The server first collects character data from content providers (such as publishing companies and game companies). This data includes images, voices, text, and 3D models of each character. The collected data is then converted into a digital format. This process utilizes high-resolution image scanning and 3D modeling software.

[0924] 2. Generating VR / AR content

[0925] The server generates VR and AR content based on digitized character data. This content can include 3D models, audio, and movement. It also designs specific algorithms and scenarios to allow users to interact with the characters.

[0926] 3. Integrating an emotion engine

[0927] The server incorporates an emotion engine that recognizes the user's emotions in real time. The emotion engine uses cameras and microphones to analyze the user's facial expressions, voice, and gestures to determine the user's emotional state. For example, if the user is smiling, it determines they are "happy," and if they are frowning, it determines they are "sad."

[0928] 4. Dynamic content changes based on emotions

[0929] The server dynamically changes the VR / AR content based on the user's emotions, as recognized by the emotion engine. For example, if the user is perceived as "happy," the character will perform more friendly actions. Conversely, if the user is perceived as "sad," the character will perform comforting actions.

[0930] 5. Content distribution

[0931] The server distributes the generated VR / AR content to the appropriate platform (such as a dedicated application, online platform, or game console). It delivers the content in a format suitable for each platform, ensuring easy access for users.

[0932] 6. User Access Management

[0933] The server verifies the user's access rights to the requested content. Users whose access rights are verified are granted access to the content, and an access log is recorded. This log includes the user ID, access date and time, and the type of content accessed.

[0934] Specific example

[0935] 1. Specific examples of character data collection and digitization

[0936] The server receives data for "Popular Character A" from the content provider. This data includes a full-body image, audio clips, and character design documents. The server then converts this data into a digital format and creates a high-resolution 3D model.

[0937] 2. Specific examples of VR / AR content generation

[0938] The server generates VR content for "Popular Character A" based on the created 3D model. Motion capture data is used to give the character realistic movements, and voice recognition functionality is added to implement user interaction.

[0939] 3. Specific Examples of Emotional Engines

[0940] The server activates an emotion engine to analyze the user's facial expressions and voice. For example, if the user is smiling, it recognizes that they are "happy," and if they are frowning, it recognizes that they are "sad."

[0941] 4. Specific examples of dynamic content changes based on emotions

[0942] The server configures the VR content's characters to perform friendly actions and speak in a way that indicates the user is "happy." Conversely, if the user is perceived as "sad," the characters' actions are adjusted to offer comforting words.

[0943] 5. Specific examples of content distribution

[0944] The server uploads the generated VR content of "Popular Character A" to a dedicated application. Users can access this content through the dedicated application and enjoy the experience on their home VR headset.

[0945] 6. Specific Examples of User Access Management

[0946] A user attempts to access VR content featuring "Popular Character A." The server verifies the user's authentication information and confirms access rights. If access is permitted, the user's access log is recorded, and their usage history for the content is managed.

[0947] This enables the system of the present invention to efficiently generate advanced character content and provide interactive experiences that respond to user emotions. This system allows character businesses in the Japanese entertainment sector to effectively expand on a new platform, providing compelling experiences to users worldwide.

[0948] The following describes the processing flow.

[0949] Step 1:

[0950] The server collects character data from content providers. This data includes character images, audio, text, and 3D models. For example, it might receive full-color illustrations of manga characters or audio data of character dialogue.

[0951] Step 2:

[0952] The server converts the received character data into a digital format. This conversion includes scanning the image data into high-resolution electronic files and generating a 3D model from the scanned data. For example, software is used to create 3D models from 2D illustrations.

[0953] Step 3:

[0954] The server generates VR and AR content based on character data converted into a digital format. Specifically, it adds movement to 3D models and implements speech recognition and generation functions to enable dialogue. Motion capture technology is used to capture natural character movements.

[0955] Step 4:

[0956] The server activates an emotion engine to analyze the user's facial expressions and voice in real time. It uses cameras and microphones to recognize the user's emotions, and the emotion engine analyzes the acquired data. For example, if the user is smiling, it determines they are "happy," and if they are frowning, it recognizes them as "sad."

[0957] Step 5:

[0958] The server dynamically modifies VR and AR content based on the recognized user's emotions. The character's actions and dialogue change according to the emotion engine's results. For example, if the user is judged to be "happy," the character will respond with a smile. If the user is judged to be "sad," the character will perform comforting actions or speak comforting words.

[0959] Step 6:

[0960] The server distributes the generated VR / AR content to the appropriate platform. It distributes the content to dedicated applications, online platforms, game consoles, etc., making it accessible to users. For example, it uploads VR content to a dedicated app, allowing users to download it.

[0961] Step 7:

[0962] A user attempts to access VR / AR content. The server checks the user's access rights and authenticates whether they have the appropriate permissions. If the user has the necessary permissions, access to the content is permitted.

[0963] Step 8:

[0964] The server records user access logs. It records which content users accessed and when, and uses this information for later analysis and operational management. For example, it stores user IDs, access dates and times, and the type of content accessed in a database.

[0965] Step 9:

[0966] Users can access permitted content. They can enjoy interactive experiences using VR headsets and AR-enabled devices. For example, they can wear a VR headset and interact with characters in a virtual space. They can also use AR devices to make characters appear in the real world.

[0967] The above outlines the specific processing flow in the system of the present invention that incorporates an emotion engine. This step-by-step processing makes it possible to manage everything from character data collection to user experience efficiently and interactively.

[0968] (Example 2)

[0969] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0970] In recent years, virtual reality (VR) and augmented reality (AR) technologies have developed rapidly, and many users are seeking interactive and engaging content. However, current systems struggle to recognize user emotions in real time and dynamically change content accordingly. As a result, the quality of the user experience is limited, and in some scenarios, there is a lack of immersion. Furthermore, the effort and resources required to support multiple platforms also remain challenges.

[0971] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0972] In this invention, the server includes means for generating character data converted into a digital format, means for generating content for virtual reality or augmented reality based on the character information converted into a digital format, means for distributing the generated content to a corresponding platform, means for verifying access rights to content requested by the user and recording access logs, an emotion recognition engine that recognizes the user's emotions in real time using a camera and microphone, and means for dynamically changing the content for virtual reality or augmented reality based on the user's emotions. This enables real-time content adjustment in response to the user's emotions, resulting in a more immersive and interactive experience.

[0973] A "digital format" is a format in which physical analog data is converted into digital data that can be used by a computer.

[0974] "Character data" refers to information about characters used in virtual reality and augmented reality content, and includes images, audio, text, 3D models, and more.

[0975] Virtual reality (VR) is a technology that allows humans to experience virtual spaces and objects created using computer technology through their five senses.

[0976] Augmented reality (AR) is a technology that overlays digital information onto real-world scenery, integrating computer-generated visual elements into the real world for display.

[0977] A "content provider" refers to a company or organization that generates and distributes digital content, and this includes publishing companies, game companies, and others.

[0978] A "platform" refers to the underlying environment or services for running and providing software and content, and includes dedicated applications, online platforms, and game consoles.

[0979] "Access rights" refer to the right of a user to access specific digital content or resources.

[0980] An "access log" is data that records the history of access to a system or database, and includes information such as the user's ID, the date and time of access, and the type of content accessed.

[0981] An "emotion recognition engine" is a software engine that uses sensors such as cameras and microphones to analyze a user's facial expressions, voice, and gestures, and recognizes their emotional state in real time.

[0982] "Dynamic modification" refers to changing the operation or output of a system in real time or on an ad-hoc basis in response to specific conditions or data.

[0983] This invention is a system that converts character data into a digital format, generates virtual reality (VR) or augmented reality (AR) content, and delivers that content to a platform. Furthermore, it incorporates an emotion recognition engine that recognizes the user's emotions and dynamically changes the content accordingly.

[0984] Character data collection and digitization

[0985] The server first collects character data from content providers (such as publishing companies and game companies). The collected data includes images, voices, text, and 3D models for each character. Next, this data is converted into digital formats using high-resolution image scanning and 3D modeling software. Specifically, image data is edited using Adobe Photoshop, converted to high-resolution images, and 3D models are generated using Autodesk Maya.

[0986] VR / AR content generation

[0987] The server generates VR and AR content based on digitized character data. This content includes 3D models, audio, and motion. Dialogue scenarios and interaction algorithms are designed to allow users to interact with the characters. Specific development utilizes game engines such as Unity and Unreal Engine, imports motion capture data, and uses the Google Cloud Speech-to-Text API for speech recognition.

[0988] Embedding an emotion recognition engine

[0989] The server incorporates an emotion recognition engine that uses cameras and microphones to recognize the user's emotions in real time. This engine analyzes the user's facial expressions using face recognition libraries such as OpenCV and Dlib, and recognizes emotions from audio data using Microsoft Azure's Emotion API.

[0990] As a concrete example, when a user connects to the system, the camera captures the user's face and analyzes their facial expressions using OpenCV. Audio data from the microphone is processed through the Emotion API, and the user's emotions are determined in real time.

[0991] Dynamic content changes based on emotions

[0992] The server dynamically modifies VR / AR content based on the user's emotions recognized by the emotion recognition engine. For example, if the user is perceived as "happy," the character will perform more friendly actions, and if the user is perceived as "sad," comforting actions will be performed. This process is carried out using an action script (e.g., Python), which calls appropriate animations and dialogue based on the specific emotional state.

[0993] Content distribution

[0994] The server delivers the generated VR / AR content to the appropriate platform. It delivers the content in a format compatible with dedicated applications, online platforms, game consoles, etc., ensuring easy access for users. Specifically, it uses a system that uploads content to an AWS S3 bucket and delivers it to users worldwide with low latency using a CDN (Content Delivery Network).

[0995] User access management

[0996] The server verifies the user's access rights to the requested content. Users who are granted access are allowed to access the content, and an access log is recorded. This log includes information such as the user ID, access date and time, and the type of content accessed. Authentication is performed using OAuth or JWT (JSON Web Token), and the access log is recorded in a database such as MySQL or MongoDB.

[0997] Specific example

[0998] 1. Specific examples of character data collection and digitization

[0999] The server downloads full-body images, audio clips, and setting materials of "Popular Character A" via the publishing company's API, digitizes the image data using Adobe Photoshop, and creates a high-resolution 3D model using Autodesk Maya.

[1000] 2. Specific examples of VR / AR content generation

[1001] The server uses the Unity engine to add movement to a 3D model of "Popular Character A," implements speech recognition functionality using the Google Cloud Speech-to-Text API, and designs user interaction scenarios.

[1002] 3. Specific Examples of Emotion Recognition Engines

[1003] The server uses OpenCV to capture the user's facial expressions in real time and the Microsoft Azure Emotion API to recognize emotions from the audio data. It determines the emotional state, such as "happy" if the user is smiling and "sad" if they are frowning.

[1004] 4. Specific examples of dynamic content changes based on emotions

[1005] If the server determines that the user is "happy," the character will smile and speak in a friendly manner; if it determines that the user is "sad," the character will offer words of comfort.

[1006] 5. Specific examples of content distribution

[1007] The server uploads VR content of "Popular Character A" generated in an AWS S3 bucket, and users can access the content through a dedicated application and experience it using their home VR headset.

[1008] 6. Specific Examples of User Access Management

[1009] The user logs into the app to access VR content featuring "Popular Character A," and the server performs user authentication using OAuth. If authentication is successful, the user's access log is recorded in a MySQL database.

[1010] This system enables real-time content adjustments based on user emotions, resulting in a more immersive and interactive experience.

[1011] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1012] Step 1: Collecting character data

[1013] The server collects character data from content providers. The input is character data (images, audio, text, 3D models) provided by publishing companies and game companies. The output is stored on the server as collected data. Specifically, the server sends API requests to content providers to retrieve the data and stores it in its own storage.

[1014] Step 2: Digitizing character data

[1015] The server converts the collected character data into a digital format. The input is the collected analog data or low-resolution digital data. The output is character data formatted as high-resolution digital data. Specifically, Adobe Photoshop is used to convert the image data to high resolution, and Autodesk Maya is used to create a 3D model.

[1016] Step 3: Generating VR / AR content

[1017] The server generates VR and AR content based on digitized character data. Inputs include digitized 3D models and audio data. Output is interactive VR / AR content. For specific actions, Unity or Unreal Engine is used to add movement to the 3D models, and the Google Cloud Speech-to-Text API is used to implement speech recognition functionality.

[1018] Step 4: Start the emotion recognition engine

[1019] The server activates an emotion recognition engine that recognizes the user's emotions in real time. The input consists of facial video and audio data sent from the user's terminal. The output is the emotion recognition result. Specifically, the server uses OpenCV and Dlib to analyze video captured from the camera, and the Microsoft Azure Emotion API to analyze audio data.

[1020] Step 5: Dynamically change content based on emotions

[1021] The server dynamically modifies VR / AR content based on the user's emotions recognized by the emotion recognition engine. The input is the emotion recognition result. The output is VR / AR content optimized according to the user's emotions. Specifically, the server uses an action script written in Python to call appropriate animations and dialogue in real time based on a specific emotional state.

[1022] Step 6: Content Distribution

[1023] The server delivers the generated VR / AR content to the corresponding platform. The input is the completed VR / AR content. The output is the content provided in a state accessible to the user. Specifically, the content is uploaded to an AWS S3 bucket and delivered to users worldwide with low latency using a CDN.

[1024] Step 7: Managing User Access

[1025] The server verifies the user's access rights to the requested content and records access logs. The input is the user's authentication information. The output is the content the user can access after authentication, and log data representing the access history. Specifically, authentication is performed using OAuth or JWT, and the results are recorded in a MySQL or MongoDB database.

[1026] This processing flow allows for the rapid and efficient delivery of interactive VR / AR experiences that respond to the user's emotions.

[1027] (Application Example 2)

[1028] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1029] Traditional virtual reality (VR) and augmented reality (AR) content systems face the challenge of providing interactive experiences that respond to user emotional changes. Furthermore, users are often limited to certain patterns of actions and scenarios when enjoying the content, resulting in a lack of realism and immersion. Additionally, there is a lack of technology to dynamically modify content based on real-time user feedback.

[1030] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for generating character data converted into a digital format, means for generating content for virtual reality or augmented reality based on the character information converted into a digital format, means for distributing the generated content to a corresponding platform, means for confirming access rights to the content requested by the user and recording access logs, means for analyzing the user's facial expressions and voice to recognize emotions, and means for dynamically changing the content based on the recognized emotions. This makes it possible to provide an interactive VR / AR experience that responds to the user's emotional state.

[1031] A "digital format" is a format in which analog data has been converted into a format that can be processed on a computer.

[1032] "Character data" refers to information such as images, audio, 3D models, and text related to a specific character.

[1033] "Virtual reality" is a technology that allows users to experience a three-dimensional artificial environment created using computer technology, as if it were reality, through their sight and hearing.

[1034] Augmented reality is a technology that overlays virtual information onto the real world, allowing users to view both the real world and digital information simultaneously.

[1035] A "platform" refers to the underlying software and hardware environment on which applications and content operate.

[1036] "Access rights" refer to the right of a user to access specific content or information, and authentication determines whether a user is entitled to access that information.

[1037] An "access log" is a record of a user's access history kept by the system, and includes information such as the user ID, the date and time of access, and the type of content accessed.

[1038] "Facial expression" refers to emotions and reactions that are expressed through the movement of the facial structure.

[1039] "Sound" refers to sounds that are transmitted as sound waves and can be heard by the human ear, and mainly includes human speech.

[1040] "Emotion" refers to the psychological response or state that a user experiences in response to a particular situation or stimulus.

[1041] "Dynamic modification" means instantly changing content and information in response to real-time changes in circumstances.

[1042] "Interactive" refers to a system that interacts with the user and responds immediately to user input.

[1043] This invention relates to a system that converts character data into a digital format, generates virtual reality (VR) or augmented reality (AR) content, and incorporates an emotion engine that recognizes user emotions and dynamically changes the content. This system is implemented using the following steps, hardware, and software.

[1044] 1. Collection and digitization of character data

[1045] The server first collects character data from content providers. This data includes images, audio, text, and 3D models of each character. This collected data is then converted into a digital format and stored. This process utilizes high-resolution image scanning and 3D modeling software.

[1046] 2. Generating VR / AR content

[1047] The server generates VR and AR content based on character data converted into a digital format. At this stage, elements such as 3D models, voice, and movement are combined. Furthermore, certain algorithms and scenarios are designed to enable interactive dialogue.

[1048] 3. Integrating an emotion engine

[1049] An emotion engine has been introduced to recognize the user's emotions in real time. Using cameras and microphones, it collects data on the user's facial expressions, voice, and gestures, and uses this data to determine their emotional state. For example, if the user is smiling, it will be determined to be "happy," and if they are frowning, it will be determined to be "sad."

[1050] 4. Dynamic content changes based on emotions

[1051] The server dynamically changes the VR / AR content based on the user's emotions, as recognized by the emotion engine. For example, if the user is perceived as "happy," the character will perform more friendly actions. Conversely, if the user is perceived as "sad," the character will perform comforting actions.

[1052] 5. Content distribution

[1053] The generated VR / AR content is delivered to the appropriate platform (such as a dedicated application, online platform, or game console). The content is delivered in a format suitable for each platform, ensuring easy access for users.

[1054] 6. User Access Management

[1055] The server checks the user's access rights to the requested content and records access logs as needed. The access logs include the user ID, access date and time, and the type of content accessed.

[1056] Hardware and software to use

[1057] This system uses the following hardware and software.

[1058] Hardware: Webcam, VR headset, smartphone

[1059] Software: Python, OpenCV, Dlib, Pyttsx3, 3D modeling software

[1060] Specific example

[1061] For example, in a system applying the present invention, suppose a user puts on a VR headset and begins interacting with "Popular Character A". If the user smiles, the animation changes to the character speaking in a friendly manner. Conversely, if the user shows a sad expression, the scenario changes to one in which the character offers words of comfort.

[1062] Example of a prompt

[1063] Please create a VR character that responds in a friendly manner when the user is smiling and in a comforting manner when the user is sad.

[1064] The necessary data is to read the user's emotions from their facial expressions and change the behavior based on that.

[1065] Use Python's OpenCV and Dlib libraries to perform emotion recognition, and the Python library pyttsx3 to provide voice feedback.

[1066] Thus, this invention provides an interactive VR / AR experience based on the user's emotions, enabling a more personalized and engaging experience for the user.

[1067] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1068] Step 1:

[1069] This system collects and digitizes character data. The server collects data such as images, audio, text, and 3D models related to characters from content providers. Input is character data, and output is character data converted to a digital format. High-resolution image scanning and 3D modeling software are used to convert the data to a digital format and save it.

[1070] Step 2:

[1071] This system generates VR / AR content. The server uses character data converted to a digital format to create VR and AR content by combining 3D models, audio, and movement. Input is character data in digital format, and output is VR / AR content. Specific algorithms and scenarios are also designed to enable interactive dialogue.

[1072] Step 3:

[1073] An emotion engine will be implemented. The server will use the emotion engine to analyze the user's facial expressions and voice to recognize their emotions. The input will be user facial expression and voice data collected by cameras and microphones, and the output will be the recognized emotional state. Data will be collected using cameras and microphones, and the user's emotions will be determined using software such as OpenCV or Dlib.

[1074] Step 4:

[1075] The content is dynamically changed according to emotions. The server dynamically changes the VR / AR content based on the user's emotions recognized by the emotion engine. The input is the recognized emotional state, and the output is the dynamically changed VR / AR content. For example, if the user is recognized as "happy," the character is set to perform friendly actions.

[1076] Step 5:

[1077] The content is delivered to the platform. The server uploads the generated VR / AR content to a dedicated application or online platform. The input is the generated VR / AR content, and the output is the content accessible on the platform. The content is delivered in an appropriate format, making it easily accessible to users.

[1078] Step 6:

[1079] This system manages user access. The server verifies user access rights to requested content and records access logs. Inputs include user requests and authentication information, while outputs include access rights verification and access logs. Users whose access rights have been verified are granted access to the content, and the system records user ID, access date and time, and the type of content accessed.

[1080] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1081] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1082] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1083] [Fourth Embodiment]

[1084] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1085] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1086] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1087] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1088] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1089] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1090] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1091] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1092] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1093] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1094] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1095] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1096] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1097] The system for carrying out the present invention will now be described. This system consists of the following main components.

[1098] 1. Character data collection and digitization

[1099] The server first receives character data from content providers (e.g., manga and anime publishers, game companies, etc.). This includes information such as character images, models, voices, and movements. The received character data is then converted into a digital format. This conversion process uses technologies such as scanning to digitize hand-drawn illustrations or digitizing 3D models.

[1100] 2. Generating VR / AR content

[1101] The server generates VR / AR content based on character information converted into a digital format. For example, it 3D models a manga character and adds movement and sound. In this generation process, algorithms for controlling character movements in real time and interactive scenarios to respond to user movements are designed.

[1102] 3. Content distribution

[1103] The server distributes the generated VR / AR content to the corresponding platform. This platform includes dedicated applications, online platforms, and game consoles. The server distributes the content in a format suitable for each platform and adjusts it to ensure smooth access for users.

[1104] 4. User Access Management

[1105] The server verifies the user's access rights to the requested content. It checks whether the user has the appropriate permissions and grants access to the content if they do. It records logs of when access is granted and uses them for later analysis and operation. This makes it possible to understand user behavior history and popular content.

[1106] Specific example

[1107] Specific examples are given below.

[1108] 1. Specific examples of character data collection and digitization

[1109] The server receives data of a "famous cartoon character" from a content provider. This data includes a full-body image of the character, audio clips, and character design documents. The server then converts this data into a digital format and creates a 3D model.

[1110] 2. Specific examples of VR / AR content generation

[1111] The server generates VR content of "famous manga characters" based on the created 3D models. Motion capture data is used to make the characters' movements appear natural. Furthermore, voice recognition functionality is incorporated to allow users to interact with the characters.

[1112] 3. Specific examples of content distribution

[1113] The server uploads the generated VR content of "famous cartoon characters" to a dedicated application. Users can download this content through the application and enjoy it on their VR headsets at home.

[1114] 4. Specific Examples of User Access Management

[1115] Users attempt to access VR content featuring "famous manga characters" through a dedicated application. The server verifies whether the user has access rights and, if access is permitted, records the user's access log. This ensures that users can enjoy the content safely and smoothly.

[1116] This invention realizes a system that efficiently and effectively handles everything from the digitization of character data to the generation and distribution of VR / AR content, as well as user access management. This will enable the development of character businesses in the Japanese entertainment sector on a new platform and make them available to users worldwide.

[1117] The following describes the processing flow.

[1118] Step 1:

[1119] The server collects character data from content providers. This character data includes a wide variety of information, such as images, audio, text, and 3D models. For example, various illustrations of manga characters and animation direction materials are provided.

[1120] Step 2:

[1121] The server converts the collected character data into a digital format. For example, it might scan hand-drawn illustrations into high-resolution image files or create character models using 3D modeling software.

[1122] Step 3:

[1123] The server generates VR (virtual reality) or AR (augmented reality) content based on character data converted into a digital format. For example, it might add movement to 3D models or implement voice recognition functionality. This creates content that allows users to interact and communicate with the characters.

[1124] Step 4:

[1125] The server distributes the generated VR / AR content to the specified platform. This distribution destination can be a dedicated application, an online platform, or a game console. For example, it might package the content into a compressed file for a dedicated application and upload it.

[1126] Step 5:

[1127] The server verifies the user's access rights. When a user attempts to access content, the server checks whether they have the appropriate permissions based on their authentication information. For example, it checks whether they are a paid member or have purchased a specific license.

[1128] Step 6:

[1129] If a user has the appropriate access rights, the server will grant them access to the content. The server records logs of users who have been granted access for later analysis and operational management. For example, it records user ID, access date and time, and the type of content accessed.

[1130] Step 7:

[1131] Users can access permitted content. They can enjoy interactive experiences using VR headsets and AR-enabled devices. For example, they can wear a VR headset to interact with cartoon characters in a virtual space, or use AR devices to make characters appear in the real world.

[1132] The above outlines the specific processing flow in the system of the present invention. This step-by-step processing makes it possible to efficiently manage everything from character data collection to user experience.

[1133] (Example 1)

[1134] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1135] In systems that generate virtual reality (VR) or augmented reality (AR) content based on existing character data, allowing users to access and experience it in real time, there is a need to streamline the content generation process and manage user access more securely and effectively. In particular, there is a lack of systems that can smoothly digitize hand-drawn illustrations and 3D models and design interactive scenarios based on them.

[1136] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1137] In this invention, the server includes means for acquiring content data from a content provider and converting it into a digital format; means for generating content for virtual reality or augmented reality based on the digitally converted content data; means for designing interactive scenarios for the generated content so that users can interact with the content in real time; means for delivering the generated virtual reality or augmented reality content to a corresponding platform; and means for verifying user access rights to the requested content and recording access logs. This makes it possible to consistently and efficiently carry out everything from content generation to delivery and user access management.

[1138] A "content provider" is an organization or individual that possesses character data to be converted into a digital format and provides that data to the server.

[1139] "Content data" refers to information necessary for virtual reality and augmented reality content, such as character images, voices, 3D models, and movements.

[1140] A "digital format" refers to a format of digital data that can be processed by a computer.

[1141] "Virtual reality" is a technology that immerses users in a computer-generated three-dimensional environment.

[1142] Augmented reality is a technology that overlays digital information onto images of the real world.

[1143] An "interactive scenario" refers to the design of a story or event that changes dynamically in response to user input and actions.

[1144] A "platform" refers to an application or online system used to deliver virtual reality or augmented reality content.

[1145] "Access rights" refer to the permissions a user has to access specific content.

[1146] An "access log" refers to a record of when a user accesses a system, and includes information such as the date and time and the content accessed.

[1147] "Hand-drawn illustrations" refer to images drawn by hand on paper or other physical media, and include converting them into a digital format.

[1148] A "3D model" refers to a three-dimensional character or object generated by a computer.

[1149] A "dedicated application" refers to specific software intended for the playback or use of virtual reality or augmented reality content.

[1150] An "online platform" refers to a website or cloud system that provides virtual reality or augmented reality content via the internet.

[1151] A "dedicated terminal" refers to a specific hardware device used to experience virtual reality or augmented reality content.

[1152] This invention relates to a system that generates virtual reality (VR) or augmented reality (AR) content based on character data, which users can access in real time. This system consists of three main elements: a server, a terminal, and a user.

[1153] Character data collection and digitization

[1154] The server receives character data from content providers. This character data includes information such as character images, 3D models, voice, and movements. The server converts the received data into a digital format. Specifically, this involves using technologies to digitize hand-drawn illustrations using high-resolution scanners, or technologies to generate 3D models using 3D scanners.

[1155] For example, we can digitize full-body images, audio clips, and character design documents of a "famous manga character" and convert them into 3D models.

[1156] VR / AR content generation

[1157] The server generates VR / AR content based on digitized character information. Specifically, it uses 3D modeling software (e.g., Blender, Maya) to fine-tune the character's 3D model and imports motion capture data to add natural movements. Furthermore, it uses speech recognition software (e.g., Google Speech-to-Text) to generate the character's voice. The server then uses the Unity engine to create a VR / AR scene using the generated 3D model and voice data.

[1158] For example, you can optimize the polygon count of a character's 3D model in Blender and then use the Unity engine to create a scene where the character walks and talks.

[1159] Content distribution

[1160] The server distributes the generated VR / AR content to the corresponding platform. Specifically, it uploads the content to a dedicated application (e.g., Android app, iOS app) and uses an API to deliver the content to the user's device in the appropriate format. If the content is updated, the server automatically distributes the new version.

[1161] For example, VR content of a "famous manga character" that has been generated can be uploaded to a dedicated application, allowing users to download this content through the app.

[1162] User access management

[1163] The server verifies the user's access rights to the requested content and records access logs. Specifically, when a user logs into a dedicated application and submits a request for content, the server compares the user's authentication information with the database to check for access rights. If access is permitted, the server provides the user with an access link to the content and records the access log.

[1164] As a concrete example, when a user attempts to access VR content featuring a "famous manga character" using a dedicated application, the server verifies the user's login information, provides an access link if they have access rights, and records an access log.

[1165] Specific examples and prompt statements

[1166] Example 1: The server downloads character data from "http: / / provider.example.com / character1.zip", unzips the ZIP file, converts it to PNG format, and saves it to the database.

[1167] Example 2: The server uses Blender to optimize a 3D model of a "famous anime character" and then creates a VR scene by adding motion capture data in Unity.

[1168] Specific example 3: The server uploads the generated VR content of "famous manga characters" to a dedicated application, and the user downloads it through the application.

[1169] Specific example 4: When a user attempts to access VR content of a "famous manga character" using a dedicated application, the server verifies the user's login information, provides an access link if they have access rights, and records an access log.

[1170] Example of a prompt:

[1171] "Generate VR / AR content based on data of famous manga characters, verify user access rights, and distribute the content."

[1172] This system enables efficient and effective digitization of character data, generation and distribution of VR / AR content, and management of user access. This allows for the development of character-based businesses in the entertainment sector on a new platform, providing engaging content to users worldwide.

[1173] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1174] Step 1:

[1175] Character data collection and digitization

[1176] The server receives character data from content providers. This character data includes image files, audio files, 3D models, etc. The server converts this data into a digital format and stores it in a dedicated database.

[1177] Specifically, the server downloads character data from a specified URL. For example, it downloads a ZIP file from "http: / / provider.example.com / character1.zip", extracts PNG image files and FBX 3D models, and then stores this data in a database.

[1178] input:

[1179] Character data (images, audio, 3D models) from content providers

[1180] Data processing:

[1181] Downloading and extracting ZIP files

[1182] Convert image files to PNG format and 3D models to FBX format.

[1183] output:

[1184] Character data converted to digital format (PNG image files, FBX 3D models, etc.)

[1185] Step 2:

[1186] VR / AR content generation

[1187] The server generates VR / AR content based on digitized character information. Specifically, it uses 3D modeling software (Blender, Maya) to fine-tune the character's 3D model and imports motion capture data to add natural movements to the character. Furthermore, it uses speech recognition software (Google Speech-to-Text) to generate the character's voice. Finally, the server uses the Unity engine to create a VR / AR scene using the generated 3D model and voice data.

[1188] Specifically, the process involves using Blender to optimize the polygon count of the character's 3D model and then using the Unity engine to create scenes where the character walks and talks.

[1189] input:

[1190] Digitized character data (PNG image files, FBX 3D models, etc.)

[1191] Data processing:

[1192] Detailed adjustments in 3D modeling software

[1193] Importing motion capture data

[1194] Generate audio data using speech recognition software

[1195] Creating VR / AR scenes with the Unity engine

[1196] output:

[1197] VR / AR content (scenes integrating 3D models, motion, and sound)

[1198] Step 3:

[1199] Content distribution

[1200] The server distributes the generated VR / AR content to the corresponding platform. Specifically, it uploads the content to a dedicated application (Android app, iOS app) and uses an API to deliver the content to the user's device in the appropriate format. If the content is updated, the server automatically distributes the new version.

[1201] Specifically, the generated VR content is uploaded to "https: / / app.example.com / updates," a dedicated application checks for the latest version of the content, and a download notification is displayed to the user.

[1202] input:

[1203] Generated VR / AR content (a scene integrating 3D models, motion, and sound)

[1204] Data processing:

[1205] Upload content

[1206] Delivery adjustment using API

[1207] output:

[1208] Content delivered to the user's device

[1209] Step 4:

[1210] User access management

[1211] The server verifies the user's access rights to the requested content and records access logs. Specifically, when a user logs into a dedicated application and submits a request for content, the server compares the user's authentication information with a database to check for access rights. If access rights are granted, the server provides an access link to the content and records access logs.

[1212] Specifically, the user logs in to a dedicated application to access VR content featuring "famous manga characters." The server verifies the user's ID and password, confirms access rights, provides an access link, and logs the access time and content.

[1213] input:

[1214] User login information (ID, password)

[1215] Content Request

[1216] Data processing:

[1217] Verification of user authentication information

[1218] Access rights verification

[1219] Access log recording

[1220] output:

[1221] Access links to content for which access rights have been verified.

[1222] Recorded access logs

[1223] (Application Example 1)

[1224] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1225] Traditionally, systems that provide character data as virtual reality (VR) or augmented reality (AR) content have struggled to efficiently integrate multiple functions such as data digitization, content generation, content distribution, and user access management. Furthermore, there was a lack of technology to effectively display characters in the user's real-world environment and provide an interactive experience. As a result, it was difficult for users to obtain a consistent AR experience.

[1226] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1227] In this invention, the server includes means for generating character data converted into a digital format, means for generating content for virtual reality or augmented reality based on the character information converted into a digital format, means for distributing the generated content to a corresponding platform, means for verifying access rights to content requested by the user and recording access logs, means for acquiring a camera feed from a smart device using augmented reality software and displaying a 3D model on the acquired camera feed, and means for playing audio that interacts with the 3D model. This enables the user to naturally display the character in the real environment and seamlessly enjoy an interactive experience.

[1228] A "digital format" refers to a form that can be stored and handled as digital data.

[1229] "Character data" refers to information about characters that appear in manga, anime, games, etc., including images, models, voices, and movements.

[1230] "Virtual reality (VR)" refers to a technology that provides interactive experiences within a virtual 3D space created using computer technology.

[1231] Augmented reality (AR) refers to a technology that overlays virtual information onto the real, physical environment.

[1232] "Content" refers to information and media (such as videos, audio, and 3D models) that users consume.

[1233] A "platform" refers to an environment, infrastructure, or service that provides the foundation for software and hardware to operate.

[1234] "Access rights" refer to permission to operate on or use specific content or services.

[1235] "Access logs" refer to data that records the history of how users have accessed a system or content.

[1236] "Augmented reality software" refers to software that overlays digital information onto the real world.

[1237] "Smart devices" refer to mobile terminals, eyeglasses, and wearable devices that have internet connectivity and high processing power.

[1238] "Camera feed" refers to real-time video footage captured by a camera.

[1239] A "3D model" refers to digital data that simulates a three-dimensional object, created using a computer.

[1240] "Audio playback means" refers to devices or functions for playing audio data through devices such as speakers or headphones.

[1241] "Interaction" refers to the two-way exchange that takes place between a user and a digital system or content.

[1242] The system realizing this invention has the function of collecting and generating character data converted into a digital format, generating virtual reality (VR) or augmented reality (AR) content based on that data, and distributing it to users. The specific configuration and operation of this system are described below.

[1243] 1. Character data collection and digitization

[1244] The server first receives character data from content providers (such as publishing companies or game companies). This data includes information such as character images, 3D models, audio clips, and motion data. This character data is then converted into a digital format using scanning and modeling technologies. This conversion enables the creation of high-quality digital content.

[1245] 2. Generating VR / AR content

[1246] The server generates VR / AR content based on character data converted into a digital format. Here, for example, 3D modeling and motion capture technology are used to create interactive characters with natural movements. Voice recognition functionality is also integrated to create characters that respond to user voices.

[1247] 3. Content distribution

[1248] The server distributes the generated VR / AR content to the corresponding platform. This platform includes dedicated applications, online platforms, and game consoles. By using smart devices (e.g., smartphones and smart glasses), users can have an interactive experience.

[1249] 4. Realization of Augmented Reality

[1250] The server uses augmented reality software to retrieve the camera feed from the smart device. A 3D model of the character is then overlaid onto the retrieved camera feed. For example, if the user is wearing smart glasses, the character will appear in the real-world scene, allowing for interaction with its movements and sounds.

[1251] 5. User Access Management

[1252] The server checks the user's access rights to the requested content and logs the access if it is granted. This log is used for later analysis and service improvement.

[1253] Explanation of specific examples

[1254] For example, suppose a user wants to enjoy an AR experience with a famous character. When the user opens a dedicated application and puts on smart glasses, the server delivers a 3D model and voice data of the character. As the user walks around in the real world, the character appears in that location and speaks and acts in response to the user's voice. Such an experience makes it feel as if the character is actually present in reality.

[1255] Example of a prompt

[1256] Design an application that provides an AR experience using a 3D model and voice data of Naruto. The application will allow users to interact with the character in real time using smart glasses. The character's movements and voice should be natural and interactive.

[1257] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1258] Step 1:

[1259] The server retrieves character data (images, 3D models, audio clips, motion data, etc.) from content providers. This data is accessed directly from the content provider's database or received as a provided file. The retrieved data is temporarily stored on the server. The input is character data, and the output is raw character data stored on the server.

[1260] Step 2:

[1261] The server converts the acquired character data into a digital format. For example, it can digitize hand-drawn illustrations using scanning technology or convert existing 2D images into 3D models. This conversion process uses specialized software (e.g., Adobe Photoshop, Blender, etc.). The input is raw character data, and the output is character data converted into a digital format.

[1262] Step 3:

[1263] The server generates VR / AR content based on character data converted into a digital format. This process uses modeling software to recreate natural character movements and incorporates voice recognition and interactive features. For example, motion capture technology is used to control character movements in real time. The input is character data in a digital format, and the output is content optimized for VR / AR.

[1264] Step 4:

[1265] The server distributes the generated VR / AR content to the corresponding platform. This platform includes dedicated applications, online platforms, and apps for smart devices, and the content needs to be converted to an appropriate format for each platform. The input is the VR / AR content, and the output is the content converted to a format suitable for the platform.

[1266] Step 5:

[1267] The device acquires the camera feed from a smart device. A 3D character model received from a server is overlaid on the acquired real-time video data. Specifically, augmented reality software is used to position the character according to real-world coordinates and enable interactive actions. The input is the camera feed and the 3D character model, and the output is the video displayed in augmented reality.

[1268] Step 6:

[1269] The device receives voice input from the user and converts it to text using a speech recognition engine (e.g., Google Speech-to-Text API). This text is then analyzed to determine how the character should react. The input is the user's voice, and the output is the text data and the character's reaction.

[1270] Step 7:

[1271] The server manages user access rights and verifies access rights to requested content. If access is granted, it logs that information and grants the user access to the content. The input is the user's request information, and the output is log data regarding access rights and the access permissions.

[1272] Step 8:

[1273] The server analyzes user access logs to understand popular content and user behavior patterns. This analysis is used to improve services and develop new content. The input is access log data, and the output is the analysis results.

[1274] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1275] This invention is a system that converts character data into a digital format, generates virtual reality (VR) or augmented reality (AR) content, and delivers that content to a platform. Furthermore, it incorporates an emotion engine that recognizes user emotions and dynamically changes the content accordingly.

[1276] 1. Character data collection and digitization

[1277] The server first collects character data from content providers (such as publishing companies and game companies). This data includes images, voices, text, and 3D models of each character. The collected data is then converted into a digital format. This process utilizes high-resolution image scanning and 3D modeling software.

[1278] 2. Generating VR / AR content

[1279] The server generates VR and AR content based on digitized character data. This content can include 3D models, audio, and movement. It also designs specific algorithms and scenarios to allow users to interact with the characters.

[1280] 3. Integrating an emotion engine

[1281] The server incorporates an emotion engine that recognizes the user's emotions in real time. The emotion engine uses cameras and microphones to analyze the user's facial expressions, voice, and gestures to determine the user's emotional state. For example, if the user is smiling, it determines they are "happy," and if they are frowning, it determines they are "sad."

[1282] 4. Dynamic content changes based on emotions

[1283] The server dynamically changes the VR / AR content based on the user's emotions, as recognized by the emotion engine. For example, if the user is perceived as "happy," the character will perform more friendly actions. Conversely, if the user is perceived as "sad," the character will perform comforting actions.

[1284] 5. Content distribution

[1285] The server distributes the generated VR / AR content to the appropriate platform (such as a dedicated application, online platform, or game console). It delivers the content in a format suitable for each platform, ensuring easy access for users.

[1286] 6. User Access Management

[1287] The server verifies the user's access rights to the requested content. Users whose access rights are verified are granted access to the content, and an access log is recorded. This log includes the user ID, access date and time, and the type of content accessed.

[1288] Specific example

[1289] 1. Specific examples of character data collection and digitization

[1290] The server receives data for "Popular Character A" from the content provider. This data includes a full-body image, audio clips, and character design documents. The server then converts this data into a digital format and creates a high-resolution 3D model.

[1291] 2. Specific examples of VR / AR content generation

[1292] The server generates VR content for "Popular Character A" based on the created 3D model. Motion capture data is used to give the character realistic movements, and voice recognition functionality is added to implement user interaction.

[1293] 3. Specific Examples of Emotional Engines

[1294] The server activates an emotion engine to analyze the user's facial expressions and voice. For example, if the user is smiling, it recognizes that they are "happy," and if they are frowning, it recognizes that they are "sad."

[1295] 4. Specific examples of dynamic content changes based on emotions

[1296] The server configures the VR content's characters to perform friendly actions and speak in a way that indicates the user is "happy." Conversely, if the user is perceived as "sad," the characters' actions are adjusted to offer comforting words.

[1297] 5. Specific examples of content distribution

[1298] The server uploads the generated VR content of "Popular Character A" to a dedicated application. Users can access this content through the dedicated application and enjoy the experience on their home VR headset.

[1299] 6. Specific Examples of User Access Management

[1300] A user attempts to access VR content featuring "Popular Character A." The server verifies the user's authentication information and confirms access rights. If access is permitted, the user's access log is recorded, and their usage history for the content is managed.

[1301] This enables the system of the present invention to efficiently generate advanced character content and provide interactive experiences that respond to user emotions. This system allows character businesses in the Japanese entertainment sector to effectively expand on a new platform, providing compelling experiences to users worldwide.

[1302] The following describes the processing flow.

[1303] Step 1:

[1304] The server collects character data from content providers. This data includes character images, audio, text, and 3D models. For example, it might receive full-color illustrations of manga characters or audio data of character dialogue.

[1305] Step 2:

[1306] The server converts the received character data into a digital format. This conversion includes scanning the image data into high-resolution electronic files and generating a 3D model from the scanned data. For example, software is used to create 3D models from 2D illustrations.

[1307] Step 3:

[1308] The server generates VR and AR content based on character data converted into a digital format. Specifically, it adds movement to 3D models and implements speech recognition and generation functions to enable dialogue. Motion capture technology is used to capture natural character movements.

[1309] Step 4:

[1310] The server activates an emotion engine to analyze the user's facial expressions and voice in real time. It uses cameras and microphones to recognize the user's emotions, and the emotion engine analyzes the acquired data. For example, if the user is smiling, it determines they are "happy," and if they are frowning, it recognizes them as "sad."

[1311] Step 5:

[1312] The server dynamically modifies VR and AR content based on the recognized user's emotions. The character's actions and dialogue change according to the emotion engine's results. For example, if the user is judged to be "happy," the character will respond with a smile. If the user is judged to be "sad," the character will perform comforting actions or speak comforting words.

[1313] Step 6:

[1314] The server distributes the generated VR / AR content to the appropriate platform. It distributes the content to dedicated applications, online platforms, game consoles, etc., making it accessible to users. For example, it uploads VR content to a dedicated app, allowing users to download it.

[1315] Step 7:

[1316] A user attempts to access VR / AR content. The server checks the user's access rights and authenticates whether they have the appropriate permissions. If the user has the necessary permissions, access to the content is permitted.

[1317] Step 8:

[1318] The server records user access logs. It records which content users accessed and when, and uses this information for later analysis and operational management. For example, it stores user IDs, access dates and times, and the type of content accessed in a database.

[1319] Step 9:

[1320] Users can access permitted content. They can enjoy interactive experiences using VR headsets and AR-enabled devices. For example, they can wear a VR headset and interact with characters in a virtual space. They can also use AR devices to make characters appear in the real world.

[1321] The above outlines the specific processing flow in the system of the present invention that incorporates an emotion engine. This step-by-step processing makes it possible to manage everything from character data collection to user experience efficiently and interactively.

[1322] (Example 2)

[1323] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1324] In recent years, virtual reality (VR) and augmented reality (AR) technologies have developed rapidly, and many users are seeking interactive and engaging content. However, current systems struggle to recognize user emotions in real time and dynamically change content accordingly. As a result, the quality of the user experience is limited, and in some scenarios, there is a lack of immersion. Furthermore, the effort and resources required to support multiple platforms also remain challenges.

[1325] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1326] In this invention, the server includes means for generating character data converted into a digital format, means for generating content for virtual reality or augmented reality based on the character information converted into a digital format, means for distributing the generated content to a corresponding platform, means for verifying access rights to content requested by the user and recording access logs, an emotion recognition engine that recognizes the user's emotions in real time using a camera and microphone, and means for dynamically changing the content for virtual reality or augmented reality based on the user's emotions. This enables real-time content adjustment in response to the user's emotions, resulting in a more immersive and interactive experience.

[1327] A "digital format" is a format in which physical analog data is converted into digital data that can be used by a computer.

[1328] "Character data" refers to information about characters used in virtual reality and augmented reality content, and includes images, audio, text, 3D models, and more.

[1329] Virtual reality (VR) is a technology that allows humans to experience virtual spaces and objects created using computer technology through their five senses.

[1330] Augmented reality (AR) is a technology that overlays digital information onto real-world scenery, integrating computer-generated visual elements into the real world for display.

[1331] A "content provider" refers to a company or organization that generates and distributes digital content, and this includes publishing companies, game companies, and others.

[1332] A "platform" refers to the underlying environment or services for running and providing software and content, and includes dedicated applications, online platforms, and game consoles.

[1333] "Access rights" refer to the right of a user to access specific digital content or resources.

[1334] An "access log" is data that records the history of access to a system or database, and includes information such as the user's ID, the date and time of access, and the type of content accessed.

[1335] An "emotion recognition engine" is a software engine that uses sensors such as cameras and microphones to analyze a user's facial expressions, voice, and gestures, and recognizes their emotional state in real time.

[1336] "Dynamic modification" refers to changing the operation or output of a system in real time or on an ad-hoc basis in response to specific conditions or data.

[1337] This invention is a system that converts character data into a digital format, generates virtual reality (VR) or augmented reality (AR) content, and delivers that content to a platform. Furthermore, it incorporates an emotion recognition engine that recognizes the user's emotions and dynamically changes the content accordingly.

[1338] Character data collection and digitization

[1339] The server first collects character data from content providers (such as publishing companies and game companies). The collected data includes images, voices, text, and 3D models for each character. Next, this data is converted into digital formats using high-resolution image scanning and 3D modeling software. Specifically, image data is edited using Adobe Photoshop, converted to high-resolution images, and 3D models are generated using Autodesk Maya.

[1340] VR / AR content generation

[1341] The server generates VR and AR content based on digitized character data. This content includes 3D models, audio, and motion. Dialogue scenarios and interaction algorithms are designed to allow users to interact with the characters. Specific development utilizes game engines such as Unity and Unreal Engine, imports motion capture data, and uses the Google Cloud Speech-to-Text API for speech recognition.

[1342] Embedding an emotion recognition engine

[1343] The server incorporates an emotion recognition engine that uses cameras and microphones to recognize the user's emotions in real time. This engine analyzes the user's facial expressions using face recognition libraries such as OpenCV and Dlib, and recognizes emotions from audio data using Microsoft Azure's Emotion API.

[1344] As a concrete example, when a user connects to the system, the camera captures the user's face and analyzes their facial expressions using OpenCV. Audio data from the microphone is processed through the Emotion API, and the user's emotions are determined in real time.

[1345] Dynamic content changes based on emotions

[1346] The server dynamically modifies VR / AR content based on the user's emotions recognized by the emotion recognition engine. For example, if the user is perceived as "happy," the character will perform more friendly actions, and if the user is perceived as "sad," comforting actions will be performed. This process is carried out using an action script (e.g., Python), which calls appropriate animations and dialogue based on the specific emotional state.

[1347] Content distribution

[1348] The server delivers the generated VR / AR content to the appropriate platform. It delivers the content in a format compatible with dedicated applications, online platforms, game consoles, etc., ensuring easy access for users. Specifically, it uses a system that uploads content to an AWS S3 bucket and delivers it to users worldwide with low latency using a CDN (Content Delivery Network).

[1349] User access management

[1350] The server verifies the user's access rights to the requested content. Users who are granted access are allowed to access the content, and an access log is recorded. This log includes information such as the user ID, access date and time, and the type of content accessed. Authentication is performed using OAuth or JWT (JSON Web Token), and the access log is recorded in a database such as MySQL or MongoDB.

[1351] Specific example

[1352] 1. Specific examples of character data collection and digitization

[1353] The server downloads full-body images, audio clips, and setting materials of "Popular Character A" via the publishing company's API, digitizes the image data using Adobe Photoshop, and creates a high-resolution 3D model using Autodesk Maya.

[1354] 2. Specific examples of VR / AR content generation

[1355] The server uses the Unity engine to add movement to a 3D model of "Popular Character A," implements speech recognition functionality using the Google Cloud Speech-to-Text API, and designs user interaction scenarios.

[1356] 3. Specific Examples of Emotion Recognition Engines

[1357] The server uses OpenCV to capture the user's facial expressions in real time and the Microsoft Azure Emotion API to recognize emotions from the audio data. It determines the emotional state, such as "happy" if the user is smiling and "sad" if they are frowning.

[1358] 4. Specific examples of dynamic content changes based on emotions

[1359] If the server determines that the user is "happy," the character will smile and speak in a friendly manner; if it determines that the user is "sad," the character will offer words of comfort.

[1360] 5. Specific examples of content distribution

[1361] The server uploads VR content of "Popular Character A" generated in an AWS S3 bucket, and users can access the content through a dedicated application and experience it using their home VR headset.

[1362] 6. Specific Examples of User Access Management

[1363] The user logs into the app to access VR content featuring "Popular Character A," and the server performs user authentication using OAuth. If authentication is successful, the user's access log is recorded in a MySQL database.

[1364] This system enables real-time content adjustments based on user emotions, resulting in a more immersive and interactive experience.

[1365] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1366] Step 1: Collecting character data

[1367] The server collects character data from content providers. The input is character data (images, audio, text, 3D models) provided by publishing companies and game companies. The output is stored on the server as collected data. Specifically, the server sends API requests to content providers to retrieve the data and stores it in its own storage.

[1368] Step 2: Digitizing character data

[1369] The server converts the collected character data into a digital format. The input is the collected analog data or low-resolution digital data. The output is character data formatted as high-resolution digital data. Specifically, Adobe Photoshop is used to convert the image data to high resolution, and Autodesk Maya is used to create a 3D model.

[1370] Step 3: Generating VR / AR content

[1371] The server generates VR and AR content based on digitized character data. Inputs include digitized 3D models and audio data. Output is interactive VR / AR content. For specific actions, Unity or Unreal Engine is used to add movement to the 3D models, and the Google Cloud Speech-to-Text API is used to implement speech recognition functionality.

[1372] Step 4: Start the emotion recognition engine

[1373] The server activates an emotion recognition engine that recognizes the user's emotions in real time. The input consists of facial video and audio data sent from the user's terminal. The output is the emotion recognition result. Specifically, the server uses OpenCV and Dlib to analyze video captured from the camera, and the Microsoft Azure Emotion API to analyze audio data.

[1374] Step 5: Dynamically change content based on emotions

[1375] The server dynamically modifies VR / AR content based on the user's emotions recognized by the emotion recognition engine. The input is the emotion recognition result. The output is VR / AR content optimized according to the user's emotions. Specifically, the server uses an action script written in Python to call appropriate animations and dialogue in real time based on a specific emotional state.

[1376] Step 6: Content Distribution

[1377] The server delivers the generated VR / AR content to the corresponding platform. The input is the completed VR / AR content. The output is the content provided in a state accessible to the user. Specifically, the content is uploaded to an AWS S3 bucket and delivered to users worldwide with low latency using a CDN.

[1378] Step 7: Managing User Access

[1379] The server verifies the user's access rights to the requested content and records access logs. The input is the user's authentication information. The output is the content the user can access after authentication, and log data representing the access history. Specifically, authentication is performed using OAuth or JWT, and the results are recorded in a MySQL or MongoDB database.

[1380] This processing flow allows for the rapid and efficient delivery of interactive VR / AR experiences that respond to the user's emotions.

[1381] (Application Example 2)

[1382] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1383] Traditional virtual reality (VR) and augmented reality (AR) content systems face the challenge of providing interactive experiences that respond to user emotional changes. Furthermore, users are often limited to certain patterns of actions and scenarios when enjoying the content, resulting in a lack of realism and immersion. Additionally, there is a lack of technology to dynamically modify content based on real-time user feedback.

[1384] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for generating character data converted into a digital format, means for generating content for virtual reality or augmented reality based on the character information converted into a digital format, means for distributing the generated content to a corresponding platform, means for confirming access rights to the content requested by the user and recording access logs, means for analyzing the user's facial expressions and voice to recognize emotions, and means for dynamically changing the content based on the recognized emotions. This makes it possible to provide an interactive VR / AR experience that responds to the user's emotional state.

[1385] A "digital format" is a format in which analog data has been converted into a format that can be processed on a computer.

[1386] "Character data" refers to information such as images, audio, 3D models, and text related to a specific character.

[1387] "Virtual reality" is a technology that allows users to experience a three-dimensional artificial environment created using computer technology, as if it were reality, through their sight and hearing.

[1388] Augmented reality is a technology that overlays virtual information onto the real world, allowing users to view both the real world and digital information simultaneously.

[1389] A "platform" refers to the underlying software and hardware environment on which applications and content operate.

[1390] "Access rights" refer to the right of a user to access specific content or information, and authentication determines whether a user is entitled to access that information.

[1391] An "access log" is a record of a user's access history kept by the system, and includes information such as the user ID, the date and time of access, and the type of content accessed.

[1392] "Facial expression" refers to emotions and reactions that are expressed through the movement of the facial structure.

[1393] "Sound" refers to sounds that are transmitted as sound waves and can be heard by the human ear, and mainly includes human speech.

[1394] "Emotion" refers to the psychological response or state that a user experiences in response to a particular situation or stimulus.

[1395] "Dynamic modification" means instantly changing content and information in response to real-time changes in circumstances.

[1396] "Interactive" refers to a system that interacts with the user and responds immediately to user input.

[1397] This invention relates to a system that converts character data into a digital format, generates virtual reality (VR) or augmented reality (AR) content, and incorporates an emotion engine that recognizes user emotions and dynamically changes the content. This system is implemented using the following steps, hardware, and software.

[1398] 1. Collection and digitization of character data

[1399] The server first collects character data from content providers. This data includes images, audio, text, and 3D models of each character. This collected data is then converted into a digital format and stored. This process utilizes high-resolution image scanning and 3D modeling software.

[1400] 2. Generating VR / AR content

[1401] The server generates VR and AR content based on character data converted into a digital format. At this stage, elements such as 3D models, voice, and movement are combined. Furthermore, certain algorithms and scenarios are designed to enable interactive dialogue.

[1402] 3. Integrating an emotion engine

[1403] An emotion engine has been introduced to recognize the user's emotions in real time. Using cameras and microphones, it collects data on the user's facial expressions, voice, and gestures, and uses this data to determine their emotional state. For example, if the user is smiling, it will be determined to be "happy," and if they are frowning, it will be determined to be "sad."

[1404] 4. Dynamic content changes based on emotions

[1405] The server dynamically changes the VR / AR content based on the user's emotions, as recognized by the emotion engine. For example, if the user is perceived as "happy," the character will perform more friendly actions. Conversely, if the user is perceived as "sad," the character will perform comforting actions.

[1406] 5. Content distribution

[1407] The generated VR / AR content is delivered to the appropriate platform (such as a dedicated application, online platform, or game console). The content is delivered in a format suitable for each platform, ensuring easy access for users.

[1408] 6. User Access Management

[1409] The server checks the user's access rights to the requested content and records access logs as needed. The access logs include the user ID, access date and time, and the type of content accessed.

[1410] Hardware and software to use

[1411] This system uses the following hardware and software.

[1412] Hardware: Webcam, VR headset, smartphone

[1413] Software: Python, OpenCV, Dlib, Pyttsx3, 3D modeling software

[1414] Specific example

[1415] For example, in a system applying the present invention, suppose a user puts on a VR headset and begins interacting with "Popular Character A". If the user smiles, the animation changes to the character speaking in a friendly manner. Conversely, if the user shows a sad expression, the scenario changes to one in which the character offers words of comfort.

[1416] Example of a prompt

[1417] Please create a VR character that responds in a friendly manner when the user is smiling and in a comforting manner when the user is sad.

[1418] The necessary data is to read the user's emotions from their facial expressions and change the behavior based on that.

[1419] Use Python's OpenCV and Dlib libraries to perform emotion recognition, and the Python library pyttsx3 to provide voice feedback.

[1420] Thus, this invention provides an interactive VR / AR experience based on the user's emotions, enabling a more personalized and engaging experience for the user.

[1421] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1422] Step 1:

[1423] This system collects and digitizes character data. The server collects data such as images, audio, text, and 3D models related to characters from content providers. Input is character data, and output is character data converted to a digital format. High-resolution image scanning and 3D modeling software are used to convert the data to a digital format and save it.

[1424] Step 2:

[1425] This system generates VR / AR content. The server uses character data converted to a digital format to create VR and AR content by combining 3D models, audio, and movement. Input is character data in digital format, and output is VR / AR content. Specific algorithms and scenarios are also designed to enable interactive dialogue.

[1426] Step 3:

[1427] An emotion engine will be implemented. The server will use the emotion engine to analyze the user's facial expressions and voice to recognize their emotions. The input will be user facial expression and voice data collected by cameras and microphones, and the output will be the recognized emotional state. Data will be collected using cameras and microphones, and the user's emotions will be determined using software such as OpenCV or Dlib.

[1428] Step 4:

[1429] The content is dynamically changed according to emotions. The server dynamically changes the VR / AR content based on the user's emotions recognized by the emotion engine. The input is the recognized emotional state, and the output is the dynamically changed VR / AR content. For example, if the user is recognized as "happy," the character is set to perform friendly actions.

[1430] Step 5:

[1431] The content is delivered to the platform. The server uploads the generated VR / AR content to a dedicated application or online platform. The input is the generated VR / AR content, and the output is the content accessible on the platform. The content is delivered in an appropriate format, making it easily accessible to users.

[1432] Step 6:

[1433] This system manages user access. The server verifies user access rights to requested content and records access logs. Inputs include user requests and authentication information, while outputs include access rights verification and access logs. Users whose access rights have been verified are granted access to the content, and the system records user ID, access date and time, and the type of content accessed.

[1434] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1435] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1436] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1437] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1438] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1439] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1440] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1441] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1442] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1443] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1444] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1445] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1446] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1447] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1448] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1449] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1450] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1451] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1452] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1453] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1454] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1455] The following is further disclosed regarding the embodiments described above.

[1456] (Claim 1)

[1457] A means for generating character data converted into a digital format,

[1458] A means for generating content for virtual reality or augmented reality based on character information converted into a digital format,

[1459] A means of delivering the generated content to the corresponding platform,

[1460] A system that includes means for verifying user access rights to requested content and recording access logs.

[1461] (Claim 2)

[1462] The system according to claim 1 for obtaining character data from a content provider.

[1463] (Claim 3)

[1464] The system according to claim 1, which uploads generated virtual reality or augmented reality content to a dedicated application or online platform.

[1465] "Example 1"

[1466] (Claim 1)

[1467] A means of obtaining content data from content providers and converting it into a digital format,

[1468] A means for generating content for virtual reality or augmented reality based on content data converted into a digital format,

[1469] A means to design interactive scenarios for generated content and enable users to interact with the content in real time,

[1470] A means for delivering generated virtual reality or augmented reality content to a corresponding platform,

[1471] A system that includes means for verifying user access rights to requested content and recording access logs.

[1472] (Claim 2)

[1473] The system according to claim 1, which obtains character data from content providers and converts handwritten illustrations and 3D models into a digital format.

[1474] (Claim 3)

[1475] The system according to claim 1, which uploads generated virtual reality or augmented reality content to a dedicated application or online platform, enabling users to enjoy the content on a dedicated device.

[1476] "Application Example 1"

[1477] (Claim 1)

[1478] A means for generating character data converted into a digital format,

[1479] A means for generating content for virtual reality or augmented reality based on character information converted into a digital format,

[1480] A means of delivering the generated content to the corresponding platform,

[1481] A means to verify access rights to content requested by users and record access logs,

[1482] A means of using augmented reality software to acquire a camera feed from a smart device and display a 3D model on the acquired camera feed,

[1483] A system that includes a means of audio playback that interacts with a 3D model.

[1484] (Claim 2)

[1485] The system according to claim 1 for obtaining character data from a content provider.

[1486] (Claim 3)

[1487] The system according to claim 1, which uploads generated virtual reality or augmented reality content to a dedicated application or online platform.

[1488] "Example 2 of combining an emotion engine"

[1489] (Claim 1)

[1490] A means for generating character data converted into a digital format,

[1491] A means for generating content for virtual reality or augmented reality based on character information converted into a digital format,

[1492] A means of delivering the generated content to the corresponding platform,

[1493] A means to verify access rights to content requested by a user and record access logs,

[1494] An emotion recognition engine that uses cameras and microphones to recognize the user's emotions in real time,

[1495] A means for dynamically changing content for virtual reality or augmented reality based on the user's emotions,

[1496] A system that includes this.

[1497] (Claim 2)

[1498] The system according to claim 1 for obtaining character data from a content provider.

[1499] (Claim 3)

[1500] The system according to claim 1, which uploads generated virtual reality or augmented reality content to a dedicated application or online platform.

[1501] "Application example 2 when combining with an emotional engine"

[1502] (Claim 1)

[1503] A means for generating character data converted into a digital format,

[1504] A means for generating content for virtual reality or augmented reality based on character information converted into a digital format,

[1505] A means of delivering the generated content to the corresponding platform,

[1506] A means to verify access rights to content requested by a user and record access logs,

[1507] A method for recognizing emotions by analyzing the user's facial expressions and voice,

[1508] Means for dynamically changing content based on recognized emotions,

[1509] A system that includes...

[1510] (Claim 2)

[1511] The system according to claim 1 for obtaining character data from a content provider.

[1512] (Claim 3)

[1513] The system according to claim 1, which uploads generated virtual reality or augmented reality content to a dedicated application or online platform. [Explanation of symbols]

[1514] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for generating character data converted into a digital format, A means for generating content for virtual reality or augmented reality based on character information converted into a digital format, A means of delivering the generated content to the corresponding platform, A system that includes means for verifying user access rights to requested content and recording access logs.

2. The system according to claim 1 for obtaining character data from a content provider.

3. The system according to claim 1, which uploads generated virtual reality or augmented reality content to a dedicated application or online platform.

Images