system

Abstract

We provide the system. [Solution] A means for analyzing digital information entered by the user and extracting components of the physical environment from the scenario, A means of creating a design for a three-dimensional structure that is automatically generated based on data accumulated in the past, A means for visually displaying three-dimensional structures generated in a virtual reality space and enabling interactive editing, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the movie production process, there is a problem that it takes a great deal of time and cost to create set designs and background art. In particular, in order to meet the detailed production and design requirements according to the movie scenario, multiple trials and errors and corrections are often required. As a result, movie producers may face delays in the production schedule and cost overruns. Furthermore, there are limited means to optimize the design before constructing a physical set, and a new method to improve the overall production efficiency is required.

Means for Solving the Problems

[0005] This invention provides a system that analyzes digital information input by a user and extracts components of a physical environment from a film script. Furthermore, it includes means for creating a three-dimensional structure design that is automatically generated based on past data, visually displaying it in a virtual reality space, and enabling interactive editing. This allows the user to review and modify the design in the virtual space, optimizing the design before physical set construction. In addition, by analyzing the editing information and updating the system's data model, the quality can be continuously improved by reflecting it in the next design generation.

[0006] "Digital information" is a general term for data expressed in a format that can be interpreted by a computer, and includes text, images, audio, video, and other digital media.

[0007] A "scenario" is a script or screenplay that outlines the structure and development of a story, as well as the actions and dialogue of the characters.

[0008] "Components of the physical environment" refer to the specific elements that make up a real space or scene, such as buildings, terrain, and decorations.

[0009] "Data accumulated in the past" refers to a collection of information and records that were previously collected and stored.

[0010] A "three-dimensional structure" is an object or design that is formed by three dimensions: length, width, and height.

[0011] A "virtual reality space" is a virtual environment created using computer technology that users can experience visually.

[0012] "Interactive editing" is a form of editing that allows users to directly manipulate and change content in real time.

[0013] A "data model" is a structure that abstracts information from a specific real world and represents the relationships between those data points. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The system of this invention aims to automatically generate a three-dimensional set design for video production based on user input, and to display and edit it in a virtual reality space. The system components and their functions are described below.

[0036] The system includes an interface for users to input scenarios and visual concepts. Users can upload scripts in text format and related reference image files. For example, they can input descriptions and settings for specific scenes in a film.

[0037] The server analyzes the input digital information and extracts key elements from the scenario. This process involves using natural language processing techniques to identify information such as location, emotional tone, and theme, and then determining the design direction based on that information.

[0038] Next, the server references a database of previously accumulated data and automatically generates a 3D structure design that fits the scenario. This process utilizes AI algorithms to enhance design quality, taking style and trends into consideration. Equally important is the ability to customize the design according to user requirements.

[0039] The generated design is incorporated into a virtual reality space for viewing on a device. Using a VR headset, users can interactively experience the design and examine its details. Furthermore, users can optimize the set design by directly adjusting object placement and changing colors and textures within the virtual reality space.

[0040] As a concrete example, consider a scenario where a user wants to create a scene for a fantasy film set in a medieval castle. The user inputs details about the castle's interior and surroundings into the scenario and uploads reference images. The server analyzes this information and automatically generates structures inspired by medieval architectural styles. The user can then view the design in a VR space and adjust wall textures and furniture placement in real time as needed.

[0041] In this way, the system of the present invention allows filmmakers to significantly reduce the time and cost of set design production and improve the efficiency of the creative process.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] Users input movie scripts and visual concepts into the system through an input interface. Text data and reference images can also be uploaded.

[0045] Step 2:

[0046] The server analyzes the digital information received from the user using natural language processing technology. It extracts important elements such as location, time period, and emotional tone from the scenario text.

[0047] Step 3:

[0048] The server automatically generates a 3D structure design based on the extracted elements and by referencing a database of previously accumulated data. It utilizes AI algorithms to construct a visual concept that takes style and historical context into account.

[0049] Step 4:

[0050] The server converts the generated 3D design into a virtual reality (VR) format. It then creates a dataset for VR rendering and prepares it for transmission to the terminal.

[0051] Step 5:

[0052] The terminal receives VR data from the server, allowing users to visually experience three-dimensional designs using a VR headset. It provides a real-time preview environment.

[0053] Step 6:

[0054] Users can view and interactively edit 3D set designs within a VR space. Specifically, they can change the placement, color, and texture of objects.

[0055] Step 7:

[0056] The server receives user edits and feedback, and performs re-analysis. The data model is updated as needed to improve the next design generation process.

[0057] (Example 1)

[0058] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0059] Traditional set design for video production has faced challenges such as being too time-consuming and costly. Furthermore, the construction of physical sets lacks flexibility, making rapid design changes and customization difficult. Additionally, traditional methods fail to efficiently utilize past designs and styles, highlighting the need for improved efficiency in the creative process.

[0060] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0061] In this invention, the server includes means for analyzing user-inputted information and extracting key environmental elements from a scenario, means for creating an automatically generated spatial design based on the collected data, and means for improving the quality of the design using an artificial intelligence algorithm. This enables efficient and flexible set design creation and rapid customization in video production.

[0062] "User-inputted information" refers to scenarios, visual concepts, and related reference images provided by users of the system.

[0063] "Analysis" is the process of understanding the content and intent based on the information entered by the user, and identifying important elements.

[0064] "Key environmental elements" refer to key information that influences the design, such as settings, locations, and emotional tones extracted from the scenario.

[0065] "Collected data" refers to the collection of information about past cases, designs, and styles accumulated within the system.

[0066] "Automatically generated spatial design" refers to the design of three-dimensional structures and virtual spaces that a system automatically generates based on input information and accumulated data.

[0067] An "artificial intelligence algorithm" is a set of advanced computational methods and procedures for data analysis and design generation, possessing learning and predictive capabilities.

[0068] "Interactive editing" refers to two-way editing operations that users can perform within a virtual space, such as rearranging objects or changing their colors and textures.

[0069] An "information model" refers to the data structure and logical representation of data used within a system.

[0070] "Reflecting a modern sensibility" means incorporating elements of past designs and styles into new designs and updating them to suit contemporary trends and styles.

[0071] The embodiments for carrying out the present invention will be described in detail below.

[0072] Users utilize the system's interface to create three-dimensional set designs necessary for video production. The user interface supports uploading text-based scenarios and related images, allowing users to input specific visual concepts. For example, a user can receive assistance in generating a concrete design by entering a prompt such as, "Generate an interior design for a medieval castle. Please use the following images as reference."

[0073] The server uses natural language processing technology and artificial intelligence algorithms to process data received from the user. At this stage, it extracts environmental elements from the scenario and automatically generates a three-dimensional space based on past design data. To improve the quality of the design, the server utilizes sophisticated computing power to provide high-quality designs that meet the user's requirements.

[0074] The generated design can be displayed on the device. The device utilizes virtual reality technology to provide an environment where the user can experience the design through a VR headset. Users can further refine the design by utilizing interactive editing functions, such as adjusting the placement of objects and changing colors and textures within the VR space.

[0075] This allows users to proceed with the set design process efficiently and flexibly, significantly reducing the time and cost involved in production.

[0076] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0077] Step 1:

[0078] The user inputs the visual concept necessary for video production through the system interface. In this step, the user provides a scenario in text format and reference images. The system then receives this data and prepares it for the next processing step. Specifically, the user clicks an upload button and selects image files from their PC's storage.

[0079] Step 2:

[0080] The server analyzes information sent by the user. Based on the input scenario, it uses natural language processing techniques to extract environmental elements and emotional tones. The input includes text data, and the output generates a list of extracted keywords and concepts. This process involves calling AI algorithms to perform the processing.

[0081] Step 3:

[0082] The server references its own database and automatically generates a 3D spatial design based on the extracted information. The AI ​​algorithm creates design proposals that consider style and trends based on the input data and past design examples. As output, a high-quality 3D design that matches the theme specified by the user is generated. The server converts this into virtual reality data to pass it on to the next process.

[0083] Step 4:

[0084] The terminal places the generated 3D design in a virtual reality space, allowing the user to view the design using a VR headset. VR content is passed to the terminal as input, and a visually immersive design space is provided as output. Specifically, the VR system outputs images to the display, allowing the user to visually confirm the design.

[0085] Step 5:

[0086] Users interactively edit designs in a VR space. As input, users select objects using the headset controllers, and the output is the change in the object's position and appearance. Specifically, users can adjust the object's position by dragging and select colors and textures from menus.

[0087] (Application Example 1)

[0088] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0089] Modern content delivery services require the efficient provision of diverse and personalized virtual environments that meet user needs. Traditional methods involve significant time and cost for manual design creation, making rapid customization to meet user requirements difficult. To address this, a system is needed that can quickly and efficiently generate and edit the virtual environments users desire.

[0090] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0091] In this invention, the server includes means for analyzing digital information input by the user and extracting components of the physical environment from the scenario; means for creating a design for a three-dimensional structure that is automatically generated based on previously accumulated data; means for visually displaying the three-dimensional structure generated in the virtual reality space and enabling interactive editing; and means for dynamically generating a three-dimensional background based on a theme selected by the user and customizing the virtual environment. This allows users to quickly and efficiently generate and edit virtual spaces according to their desired themes, enabling the provision of more advanced and personalized content experiences.

[0092] A "user" is an individual or group that uses the system to input digital information and create and edit virtual environments.

[0093] "Digital information" refers to text-based data and image materials, including scenarios and visual concepts, that users input into the system.

[0094] "Analysis" is the process of processing input digital information to extract important elements and features of a scenario.

[0095] "Components of the physical environment" refers to the specific elements and attributes of the virtual reality space defined based on the scenario, and is a general term that includes buildings, terrain, and so on.

[0096] A "three-dimensional structure" is a three-dimensional object or building generated within a virtual reality space.

[0097] "Automatically generated" refers to a process that is created autonomously by a system under program control, without requiring human intervention.

[0098] "Interactive editing" refers to the ability for users to change the placement and attributes of objects in a virtual reality space in real time.

[0099] "Dynamic generation" refers to a process where content is generated while instantly changing in response to user input and circumstances.

[0100] A "virtual space" is an artificial three-dimensional space constructed by a computer, in which users can interact through an interface.

[0101] To implement this invention, three main elements are required: a server, a terminal, and a user.

[0102] server:

[0103] The server receives digital information input from the user and performs analysis using natural language processing technology. Specifically, it uses a natural language processing library implemented in Python to extract the components of the physical environment from the scenario. Based on this information, a generative AI model operating internally using TENSORFLOW® automatically generates an appropriate three-dimensional structure. As a result, the generated three-dimensional design can be customized according to the user's requirements.

[0104] Terminal:

[0105] The device runs an application created using Unity, visually displaying three-dimensional structures generated in a virtual reality space to the user. By using an interface such as a VR headset or smart glasses, interactive editing of each object becomes possible. Users can directly move objects and change their attributes within this space, allowing for visual and intuitive design optimization.

[0106] User:

[0107] Users have the ability to input scenarios and visual concepts through the provided interface and instantly generate and edit the necessary 3D backgrounds. For example, if a user selects the "Summer Beach" theme and specifies certain elements as prompts, the system automatically creates a virtual environment that matches that theme. Through this process, users can obtain a richer virtual experience.

[0108] Specific example:

[0109] If a user wants to create a unique background for an online event, the system will generate a detailed 3D background based on the entered theme, including, for example, a coastline or a starry night sky. This generation process is done in real time, allowing users to instantly see the results and edit them as needed.

[0110] Example of a prompt:

[0111] "User-selected theme: Summer Beach. Generate a 3D background with related textures and sounds."

[0112] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0113] Step 1:

[0114] Users input scenarios and visual concepts through the system interface. This input data consists of text-based scenarios and associated image files. This data is then sent to the server as basic information to express the user's creative intent.

[0115] Step 2:

[0116] The server analyzes the received digital information using natural language processing techniques. Here, a natural language processing library implemented in Python is used to extract the components and main themes of the physical environment from text data. The input consists of text and images provided by the user, and the output is the extracted component and theme information. This output forms the basis for 3D design generation.

[0117] Step 3:

[0118] Based on the analysis results, the server references previously accumulated data and utilizes a generated AI model to design an appropriate three-dimensional structure. In this process, an initial three-dimensional model is generated based on the analyzed theme and style information. The input is the theme information and accumulated data obtained in step 2, and the output is the automatically generated three-dimensional design.

[0119] Step 4:

[0120] The 3D structure design generated by the server is sent to the terminal and placed in a virtual reality space by an application using Unity. The terminal then displays the design visually to the user through a VR headset or smart glasses. The input is the generated 3D design, and the output is the virtual space visualized by the user.

[0121] Step 5:

[0122] Users interactively edit designs within a virtual reality space. Specifically, they can adjust the position and attributes of objects in real time via their device. The virtual space obtained in the previous step is the input, and the output is the final design after the user's edits.

[0123] Step 6:

[0124] After editing is complete, the server analyzes the user's editing information and updates the system's data model. This ensures that the user's preferences and style are reflected in subsequent design generation. The input is the user's editing history, and the output is the updated data model.

[0125] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0126] The present invention provides a system that recognizes a user's emotions in real time and dynamically adjusts the design and environment of a virtual reality space based on those emotions. Specific embodiments of the present invention are described below.

[0127] The system includes an interface for users to input scenarios and visual concepts. Users can upload text information and image materials related to movie scenes, providing basic design information. For example, they can specify elements that correspond to the scene setting or the characters' psychology.

[0128] The server analyzes the input digital information and extracts the necessary components. Using natural language processing technology, it identifies elements such as location and emotional tone from the scenario and determines the direction of the generated design.

[0129] The emotion engine utilizes devices and sensors to recognize the user's emotions in real time. It records the user's facial expressions, voice tone, and heart rate, and analyzes their emotions using AI algorithms. This emotional data is then used to dynamically influence the design.

[0130] The server uses data provided by the emotion engine to modify the visual concept of the three-dimensional structure. If the user is feeling tense, the system can adjust the design to create a more relaxed atmosphere. Conversely, if a lively scene is desired, it will provide a more vibrant and energetic design.

[0131] The generated design is converted into a virtual reality format and delivered to the user in real time via the device. The user can use a VR headset to visually check the design in a three-dimensional space and interactively edit it while moving around the environment.

[0132] As a concrete example, consider a situation where a user needs to create a sad scene. When the emotion engine recognizes the user's emotions and detects sadness, the server darkens the entire environment and alters the sound effects to enhance the atmosphere. Such real-time adjustments allow filmmakers to efficiently create emotionally rich scenes.

[0133] This invention allows filmmakers to gain emotional insights into set design and construct scenes that evoke a deep emotional connection with the audience.

[0134] The following describes the processing flow.

[0135] Step 1:

[0136] Users input film scripts and visual concepts through the interface. They upload script text and related images to the system, providing foundational information to concretize the design direction.

[0137] Step 2:

[0138] The server analyzes the digital information received from the user. Using natural language processing technology, it extracts elements such as location, historical context, and emotional tone from the scenario to derive design directions.

[0139] Step 3:

[0140] The server acquires user emotional data through an emotion engine. The emotion engine processes biometric information such as the user's facial expressions, voice, and heart rate using an emotion analysis algorithm to identify their emotional state.

[0141] Step 4:

[0142] The server determines the design of the 3D structure based on extracted scenario elements and user emotion data. It dynamically adjusts the colors, lighting, and details of the environment and structure to match the user's emotions.

[0143] Step 5:

[0144] The server converts the generated design into a virtual reality format and prepares it to be delivered to the user via the terminal.

[0145] Step 6:

[0146] The device displays a three-dimensional design in a virtual reality space in real time using the user's VR headset. The user visually confirms the design in the VR environment and receives feedback that reflects their emotions.

[0147] Step 7:

[0148] Users can perform interactive editing directly within the VR space. They can optimize the design by adjusting object placement and environmental changes as needed.

[0149] Step 8:

[0150] The server receives user edits and emotional responses and updates its data model. The server then uses this information to refine future design generation.

[0151] (Example 2)

[0152] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0153] Current virtual space design systems struggle to reflect users' emotional states in real time and dynamically adjust environmental elements, creating a need for methods to provide more emotionally immersive experiences in the fields of entertainment and education. Furthermore, there is a lack of technical means to quickly and intuitively edit designs and reflect those changes throughout the entire system.

[0154] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0155] In this invention, the server includes means for analyzing user input and extracting environmental elements from a scenario; means for analyzing the user's emotional state in real time using an emotion recognition device and dynamically adjusting the environmental elements based on that analysis; and means for visually displaying the dynamically changing environment in the virtual space and enabling interactive editing. This makes it possible to generate an attractive virtual environment that reflects the user's emotions and to edit the design interactively.

[0156] "User-inputted information" refers to text and image data related to the scenario or visual concept that the user provides through the interface.

[0157] "Means for extracting environmental elements" refers to methods for analyzing physical or emotional characteristics from input information and identifying the necessary components for a virtual space.

[0158] An "emotion recognition device" is a general term for hardware and software that acquires biometric information such as the user's facial expressions, voice tone, and heart rate, and analyzes the user's emotional state in real time.

[0159] "Dynamic adjustment methods" refer to methods for creating an environment that responds to the user's emotions by changing the design and elements of the virtual environment based on emotional data obtained in real time.

[0160] A "virtual space" is a three-dimensional digital environment generated by a computer, a simulation in which users can have an immersive experience through sight and interaction.

[0161] "Means of enabling two-way editing" refers to technologies that allow users to change designs and settings within a virtual space in real time, and for those changes to be immediately reflected in the system.

[0162] This invention is a system that dynamically adjusts the design and environment of a virtual space based on the user's emotions. The system consists primarily of a user, a server, and a terminal. First, the user inputs information related to the scenario and visual concept through an interface. The interface includes text input forms and image upload functions, allowing for detailed scene configuration.

[0163] The server receives information provided by the user and analyzes it using natural language processing technology. This analysis extracts environmental elements from the scenario and prepares the system for utilizing emotional data acquired in real time by the user's emotion recognition device. The emotion recognition device integrates cameras and biometric sensors to monitor the user's facial expressions, voice tone, and heart rate. This data is analyzed by an AI algorithm to identify the user's emotional state.

[0164] The server dynamically adjusts environmental elements within the virtual space based on emotional data. For example, if the server determines that the user is relaxed, it generates a design with soft colors and soothing music. The generated environment is then converted into a virtual reality format and delivered to the user in real time via the terminal.

[0165] The terminal allows users to visually experience a virtual space using devices such as VR headsets. Users can interactively edit within this virtual space, and those changes are immediately reflected in the system. For example, if a user wants to create a sad scene, the emotion engine detects that emotion, and the server adjusts the environment to a darker color scheme and changes the music to emphasize the sad atmosphere.

[0166] An example of a prompt to input into the generation AI model might be, "If the user is feeling anxious, generate a calm design that matches that emotion." This system allows users to flexibly and effectively design virtual environments and provide emotionally rich experiences.

[0167] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0168] Step 1:

[0169] Users input scenario information through an interface. The interface includes text input forms and image upload functions, allowing users to provide text and images related to movie scenes. The data obtained as input is sent to the server for analysis.

[0170] Step 2:

[0171] The server analyzes the received input data and extracts environmental elements from the scenario using natural language processing techniques. The input here is the scenario information provided by the user, and the output is a list of environmental elements necessary for initial setup of the virtual space. Specifically, it performs keyword extraction and sentiment tone analysis from the text information.

[0172] Step 3:

[0173] The emotion recognition device collects real-time emotional data from the user. Inputs include the user's facial expressions, voice tone, and heart rate. This data is processed by an emotion analysis algorithm, which outputs the user's current emotional state (e.g., joy, sadness, tension).

[0174] Step 4:

[0175] The server dynamically adjusts the virtual space design using emotion data obtained from emotion recognition. Inputs include a list of environment elements and the user's emotional state, while output is the adjusted virtual space design. Specific actions include changing the environment's color scheme, selecting music, and applying dynamic background effects.

[0176] Step 5:

[0177] The terminal converts the adjusted virtual space into VR format and provides it to the user in real time. The user can visually experience the virtual space and move freely using a VR headset. The input here is adjusted design data from the server, and the output is a visualized three-dimensional virtual environment.

[0178] Step 6:

[0179] Users can interactively edit within the virtual space and have their changes reflected instantly. User input is sent to the server as edited content, and the updated design is output. This includes actions such as rearranging objects and readjusting color tones.

[0180] (Application Example 2)

[0181] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0182] In virtual reality viewing experiences, it is difficult to dynamically adjust the environment in real time according to the user's emotional state. Therefore, there is a challenge in achieving sufficient emotional resonance between the content the user is viewing and their emotional state at that time.

[0183] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0184] In this invention, the server includes means for recognizing the user's emotional state and dynamically adjusting visual and acoustic elements within the virtual environment based on acquired emotional data; means for analyzing digital information input by the user and extracting components of the physical environment from the scenario; and means for creating a design for a three-dimensional structure that is automatically generated based on previously accumulated data. This enables real-time adjustment of the viewing environment in accordance with the user's emotions.

[0185] "Emotional state" refers to the user's current psychological tendency, identified based on physiological and psychological data such as the user's facial expressions, voice, and heart rate.

[0186] A "virtual environment" is a digital space created by a computer that provides a visual and auditory experience in a place where the user is not physically present.

[0187] "Visual elements" refer to visual content such as colors, shapes, and movements that are presented to the user within a virtual environment.

[0188] "Audio elements" refer to audio content such as sounds, music, and sound effects presented to the user within a virtual environment.

[0189] "Dynamic adjustment" refers to updating and changing the environment in real time according to the user's emotional state.

[0190] "Digital information" refers to electronic data that users provide to the system, such as movie scenes, images, and text documents.

[0191] "Components of the physical environment" refer to the visual and physical characteristics that are extracted from digital information and should be reproduced within the virtual environment.

[0192] A "three-dimensional structure" refers to an object that is generated three-dimensionally within a virtual environment and can be visually perceived by the user.

[0193] To realize this invention, a system is needed that recognizes the user's emotional state in real time and dynamically adjusts the virtual reality space accordingly. The outline of this system is described below.

[0194] The server utilizes high-performance computing devices and leverages multiple digital information processing middleware. It analyzes movie scenes and text information provided by the user through the interface using natural language processing techniques. Here, Google® Cloud Natural Language API is used to extract physical environment components and emotional tones from the scenario. Based on this analysis, the server uses a generative AI model to concretize the virtual environment design and generates a three-dimensional structure using Unity.

[0195] The user's emotional state is detected by a head-mounted display and physiological sensors, and analyzed by a program utilizing the Microsoft® Azure® Emotion API. The acquired emotional data is sent to a server to adjust the visual and auditory elements within the virtual reality space, causing the environment to change in real time.

[0196] This system allows for adjustments to visual and auditory elements in response to the user's emotional state, such as tension or relaxation. For example, if a user is relaxed while watching a movie, the server will change the scene to warmer colors and provide soothing music. This dynamic adjustment allows the user to experience a deeper sense of immersion.

[0197] As an example of a prompt, you could enter instructions such as, "Provide emotional visual and sound changes based on a movie scene. If the user is relaxed, select a bright and relaxing soundtrack for the scenery."

[0198] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0199] Step 1:

[0200] Users input movie scenes and text information into the system via a terminal. The entered digital information is sent to the server as initial data. The server receives this data and prepares it for analysis.

[0201] Step 2:

[0202] The server uses the Google Cloud Natural Language API to analyze the digital information it receives. Specifically, it extracts elements of the physical environment and emotional tone from the scenario. It processes the scenario as input data and outputs it as "emotional tone" and "environmental elements."

[0203] Step 3:

[0204] The server uses a generative AI model to create a virtual environment design based on the analysis of emotional tone. This AI model automatically generates the optimal environment design based on past database data and user-specified prompts. The generated design data is output as three-dimensional structure data.

[0205] Step 4:

[0206] The user's emotional state is detected through a head-mounted display and physiological sensors attached to the device. The acquired emotional data is immediately analyzed using the Microsoft Azure Emotion API. The "user's emotional state" is extracted from this raw emotional data.

[0207] Step 5:

[0208] The server dynamically constructs a virtual reality environment through the Unity engine, utilizing the user's emotional state and 3D structure data. Visual and acoustic elements are adjusted in real time according to the emotional state, providing the user with a visually and acoustically optimized experience.

[0209] Step 6:

[0210] Users experience a tuned virtual environment through a head-mounted display. The visualized environment is interactive, allowing users to move freely and interact directly with it. This feedback is sent to the server in real time, and the data model is updated to inform future environment adjustments.

[0211] By coordinating each step, an interactive and real-time virtual reality environment based on the user's emotions is realized.

[0212] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0213] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0214] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0215] [Second Embodiment]

[0216] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0217] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0218] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0219] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0220] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0221] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0222] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0223] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0224] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0225] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0226] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0227] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0228] The system of this invention aims to automatically generate a three-dimensional set design for video production based on user input, and to display and edit it in a virtual reality space. The system components and their functions are described below.

[0229] The system includes an interface for users to input scenarios and visual concepts. Users can upload scripts in text format and related reference image files. For example, they can input descriptions and settings for specific scenes in a film.

[0230] The server analyzes the input digital information and extracts key elements from the scenario. This process involves using natural language processing techniques to identify information such as location, emotional tone, and theme, and then determining the design direction based on that information.

[0231] Next, the server references a database of previously accumulated data and automatically generates a 3D structure design that fits the scenario. This process utilizes AI algorithms to enhance design quality, taking style and trends into consideration. Equally important is the ability to customize the design according to user requirements.

[0232] The generated design is incorporated into a virtual reality space for viewing on a device. Using a VR headset, users can interactively experience the design and examine its details. Furthermore, users can optimize the set design by directly adjusting object placement and changing colors and textures within the virtual reality space.

[0233] As a concrete example, consider a scenario where a user wants to create a scene for a fantasy film set in a medieval castle. The user inputs details about the castle's interior and surroundings into the scenario and uploads reference images. The server analyzes this information and automatically generates structures inspired by medieval architectural styles. The user can then view the design in a VR space and adjust wall textures and furniture placement in real time as needed.

[0234] In this way, the system of the present invention allows filmmakers to significantly reduce the time and cost of set design production and improve the efficiency of the creative process.

[0235] The following describes the processing flow.

[0236] Step 1:

[0237] Users input movie scripts and visual concepts into the system through an input interface. Text data and reference images can also be uploaded.

[0238] Step 2:

[0239] The server analyzes the digital information received from the user using natural language processing technology. It extracts important elements such as location, time period, and emotional tone from the scenario text.

[0240] Step 3:

[0241] The server automatically generates a 3D structure design based on the extracted elements and by referencing a database of previously accumulated data. It utilizes AI algorithms to construct a visual concept that takes style and historical context into account.

[0242] Step 4:

[0243] The server converts the generated 3D design into a virtual reality (VR) format. It then creates a dataset for VR rendering and prepares it for transmission to the terminal.

[0244] Step 5:

[0245] The terminal receives VR data from the server, allowing users to visually experience three-dimensional designs using a VR headset. It provides a real-time preview environment.

[0246] Step 6:

[0247] Users can view and interactively edit 3D set designs within a VR space. Specifically, they can change the placement, color, and texture of objects.

[0248] Step 7:

[0249] The server receives user edits and feedback, and performs re-analysis. The data model is updated as needed to improve the next design generation process.

[0250] (Example 1)

[0251] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0252] Traditional set design for video production has faced challenges such as being too time-consuming and costly. Furthermore, the construction of physical sets lacks flexibility, making rapid design changes and customization difficult. Additionally, traditional methods fail to efficiently utilize past designs and styles, highlighting the need for improved efficiency in the creative process.

[0253] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0254] In this invention, the server includes means for analyzing user-inputted information and extracting key environmental elements from a scenario, means for creating an automatically generated spatial design based on the collected data, and means for improving the quality of the design using an artificial intelligence algorithm. This enables efficient and flexible set design creation and rapid customization in video production.

[0255] "User-inputted information" refers to scenarios, visual concepts, and related reference images provided by users of the system.

[0256] "Analysis" is the process of understanding the content and intent based on the information entered by the user, and identifying important elements.

[0257] "Key environmental elements" refer to key information that influences the design, such as settings, locations, and emotional tones extracted from the scenario.

[0258] "Collected data" refers to the collection of information about past cases, designs, and styles accumulated within the system.

[0259] "Automatically generated spatial design" refers to the design of three-dimensional structures and virtual spaces that a system automatically generates based on input information and accumulated data.

[0260] An "artificial intelligence algorithm" is a set of advanced computational methods and procedures for data analysis and design generation, possessing learning and predictive capabilities.

[0261] "Interactive editing" refers to two-way editing operations that users can perform within a virtual space, such as rearranging objects or changing their colors and textures.

[0262] An "information model" refers to the data structure and logical representation of data used within a system.

[0263] "Reflecting a modern sensibility" means incorporating elements of past designs and styles into new designs and updating them to suit contemporary trends and styles.

[0264] The embodiments for carrying out the present invention will be described in detail below.

[0265] Users utilize the system's interface to create three-dimensional set designs necessary for video production. The user interface supports uploading text-based scenarios and related images, allowing users to input specific visual concepts. For example, a user can receive assistance in generating a concrete design by entering a prompt such as, "Generate an interior design for a medieval castle. Please use the following images as reference."

[0266] The server uses natural language processing technology and artificial intelligence algorithms to process data received from the user. At this stage, it extracts environmental elements from the scenario and automatically generates a three-dimensional space based on past design data. To improve the quality of the design, the server utilizes sophisticated computing power to provide high-quality designs that meet the user's requirements.

[0267] The generated design can be displayed on the device. The device utilizes virtual reality technology to provide an environment where the user can experience the design through a VR headset. Users can further refine the design by utilizing interactive editing functions, such as adjusting the placement of objects and changing colors and textures within the VR space.

[0268] This allows users to proceed with the set design process efficiently and flexibly, significantly reducing the time and cost involved in production.

[0269] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0270] Step 1:

[0271] The user inputs the visual concept necessary for video production through the system interface. In this step, the user provides a scenario in text format and reference images. The system then receives this data and prepares it for the next processing step. Specifically, the user clicks an upload button and selects image files from their PC's storage.

[0272] Step 2:

[0273] The server analyzes information sent by the user. Based on the input scenario, it uses natural language processing techniques to extract environmental elements and emotional tones. The input includes text data, and the output generates a list of extracted keywords and concepts. This process involves calling AI algorithms to perform the processing.

[0274] Step 3:

[0275] The server references its own database and automatically generates a 3D spatial design based on the extracted information. The AI ​​algorithm creates design proposals that consider style and trends based on the input data and past design examples. As output, a high-quality 3D design that matches the theme specified by the user is generated. The server converts this into virtual reality data to pass it on to the next process.

[0276] Step 4:

[0277] The terminal places the generated 3D design in a virtual reality space, allowing the user to view the design using a VR headset. VR content is passed to the terminal as input, and a visually immersive design space is provided as output. Specifically, the VR system outputs images to the display, allowing the user to visually confirm the design.

[0278] Step 5:

[0279] Users interactively edit designs in a VR space. As input, users select objects using the headset controllers, and the output is the change in the object's position and appearance. Specifically, users can adjust the object's position by dragging and select colors and textures from menus.

[0280] (Application Example 1)

[0281] Next, Application Example 1 will be described. In the following description, the data processing apparatus 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0282] In modern content distribution services, it is required to efficiently provide diverse and personalized virtual environments demanded by users. In conventional methods, the time and cost required for manual design creation are large, and it is difficult to quickly customize according to user requests. To solve this problem, a system that can quickly and efficiently generate and edit the virtual environment demanded by users is necessary.

[0283] The specific processing by the specific processing unit 290 of the data processing apparatus 12 in Application Example 1 is realized by the following means.

[0284] In this invention, the server includes means for analyzing digital information input by a user and extracting components of the physical environment from a scenario, means for creating a design of a three-dimensional structure automatically generated based on data accumulated in the past, means for visually displaying the three-dimensional structure generated in the virtual reality space and enabling interactive editing, and means for dynamically generating a three-dimensional background based on a theme selected by the user and customizing the virtual environment. Thereby, the user can quickly and efficiently generate and edit a virtual space according to his desired theme, and it becomes possible to provide a more advanced and personalized content experience.

[0285] A "user" is an individual or group that uses the system to input digital information and generate / edit a virtual environment.

[0286] "Digital information" is text-form data and image materials including scenarios and visual concepts input by the user to the system.

[0287] "Analyze" is a process of processing the input digital information to extract important elements and features of the scenario.

[0288] "Components of the physical environment" refers to the specific elements and attributes of the virtual reality space defined based on the scenario, and is a general term that includes buildings, terrain, and so on.

[0289] A "three-dimensional structure" is a three-dimensional object or building generated within a virtual reality space.

[0290] "Automatically generated" refers to a process that is created autonomously by a system under program control, without requiring human intervention.

[0291] "Interactive editing" refers to the ability for users to change the placement and attributes of objects in a virtual reality space in real time.

[0292] "Dynamic generation" refers to a process where content is generated while instantly changing in response to user input and circumstances.

[0293] A "virtual space" is an artificial three-dimensional space constructed by a computer, in which users can interact through an interface.

[0294] To implement this invention, three main elements are required: a server, a terminal, and a user.

[0295] server:

[0296] The server receives digital information input from the user and performs analysis using natural language processing techniques. Specifically, it uses a natural language processing library implemented in Python to extract the components of the physical environment from the scenario. Based on this information, a generative AI model operating internally using TensorFlow automatically generates appropriate three-dimensional structures. As a result, the generated three-dimensional design can be customized according to the user's requirements.

[0297] Terminal:

[0298] The device runs an application created using Unity, visually displaying three-dimensional structures generated in a virtual reality space to the user. By using an interface such as a VR headset or smart glasses, interactive editing of each object becomes possible. Users can directly move objects and change their attributes within this space, allowing for visual and intuitive design optimization.

[0299] User:

[0300] Users have the ability to input scenarios and visual concepts through the provided interface and instantly generate and edit the necessary 3D backgrounds. For example, if a user selects the "Summer Beach" theme and specifies certain elements as prompts, the system automatically creates a virtual environment that matches that theme. Through this process, users can obtain a richer virtual experience.

[0301] Specific example:

[0302] If a user wants to create a unique background for an online event, the system will generate a detailed 3D background based on the entered theme, including, for example, a coastline or a starry night sky. This generation process is done in real time, allowing users to instantly see the results and edit them as needed.

[0303] Example of a prompt:

[0304] "User-selected theme: Summer Beach. Generate a 3D background with related textures and sounds."

[0305] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0306] Step 1:

[0307] The user inputs scenarios and visual concepts through the system interface. At this time, the input data is a scenario in text format and related image files. These data are sent to the server as basic information for expressing the user's creative intention.

[0308] Step 2:

[0309] The server analyzes the received digital information using natural language processing technology. Here, a natural language processing library implemented in Python is used to extract the components of the physical environment and the main themes from the text data. The input is the text and images provided by the user, and the output is the extracted component and theme information. This output serves as the basis for 3D design generation.

[0310] Step 3:

[0311] Based on the analysis results, the server refers to the data accumulated in the past and utilizes the generative AI model to design an appropriate 3D structure. In this process, an initial 3D model is generated based on the analyzed themes and style information. The input is the theme information obtained in Step 2 and the accumulated data, and the output is the automatically generated 3D design.

[0312] Step 4:

[0313] The design of the 3D structure generated by the server is sent to the terminal and placed in the virtual reality space by an application using Unity. Here, the terminal visually displays the design to the user through a VR headset or smart glasses. The input is the generated 3D design, and the output is the virtual space visualized by the user.

[0314] Step 5:

[0315] Users interactively edit designs within a virtual reality space. Specifically, they can adjust the position and attributes of objects in real time via their device. The virtual space obtained in the previous step is the input, and the output is the final design after the user's edits.

[0316] Step 6:

[0317] After editing is complete, the server analyzes the user's editing information and updates the system's data model. This ensures that the user's preferences and style are reflected in subsequent design generation. The input is the user's editing history, and the output is the updated data model.

[0318] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0319] The present invention provides a system that recognizes a user's emotions in real time and dynamically adjusts the design and environment of a virtual reality space based on those emotions. Specific embodiments of the present invention are described below.

[0320] The system includes an interface for users to input scenarios and visual concepts. Users can upload text information and image materials related to movie scenes, providing basic design information. For example, they can specify elements that correspond to the scene setting or the characters' psychology.

[0321] The server analyzes the input digital information and extracts the necessary components. Using natural language processing technology, it identifies elements such as location and emotional tone from the scenario and determines the direction of the generated design.

[0322] The emotion engine utilizes devices and sensors to recognize the user's emotions in real time. It records the user's facial expressions, voice tone, and heart rate, and analyzes their emotions using AI algorithms. This emotional data is then used to dynamically influence the design.

[0323] The server uses data provided by the emotion engine to modify the visual concept of the three-dimensional structure. If the user is feeling tense, the system can adjust the design to create a more relaxed atmosphere. Conversely, if a lively scene is desired, it will provide a more vibrant and energetic design.

[0324] The generated design is converted into a virtual reality format and delivered to the user in real time via the device. The user can use a VR headset to visually check the design in a three-dimensional space and interactively edit it while moving around the environment.

[0325] As a concrete example, consider a situation where a user needs to create a sad scene. When the emotion engine recognizes the user's emotions and detects sadness, the server darkens the entire environment and alters the sound effects to enhance the atmosphere. Such real-time adjustments allow filmmakers to efficiently create emotionally rich scenes.

[0326] This invention allows filmmakers to gain emotional insights into set design and construct scenes that evoke a deep emotional connection with the audience.

[0327] The following describes the processing flow.

[0328] Step 1:

[0329] Users input film scripts and visual concepts through the interface. They upload script text and related images to the system, providing foundational information to concretize the design direction.

[0330] Step 2:

[0331] The server analyzes the digital information received from the user. Using natural language processing technology, it extracts elements such as location, historical context, and emotional tone from the scenario to derive design directions.

[0332] Step 3:

[0333] The server acquires user emotional data through an emotion engine. The emotion engine processes biometric information such as the user's facial expressions, voice, and heart rate using an emotion analysis algorithm to identify their emotional state.

[0334] Step 4:

[0335] The server determines the design of the 3D structure based on extracted scenario elements and user emotion data. It dynamically adjusts the colors, lighting, and details of the environment and structure to match the user's emotions.

[0336] Step 5:

[0337] The server converts the generated design into a virtual reality format and prepares it to be delivered to the user via the terminal.

[0338] Step 6:

[0339] The device displays a three-dimensional design in a virtual reality space in real time using the user's VR headset. The user visually confirms the design in the VR environment and receives feedback that reflects their emotions.

[0340] Step 7:

[0341] Users can perform interactive editing directly within the VR space. They can optimize the design by adjusting object placement and environmental changes as needed.

[0342] Step 8:

[0343] The server receives user edits and emotional responses and updates its data model. The server then uses this information to refine future design generation.

[0344] (Example 2)

[0345] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0346] Current virtual space design systems struggle to reflect users' emotional states in real time and dynamically adjust environmental elements, creating a need for methods to provide more emotionally immersive experiences in the fields of entertainment and education. Furthermore, there is a lack of technical means to quickly and intuitively edit designs and reflect those changes throughout the entire system.

[0347] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0348] In this invention, the server includes means for analyzing user input and extracting environmental elements from a scenario; means for analyzing the user's emotional state in real time using an emotion recognition device and dynamically adjusting the environmental elements based on that analysis; and means for visually displaying the dynamically changing environment in the virtual space and enabling interactive editing. This makes it possible to generate an attractive virtual environment that reflects the user's emotions and to edit the design interactively.

[0349] "User-inputted information" refers to text and image data related to the scenario or visual concept that the user provides through the interface.

[0350] "Means for extracting environmental elements" refers to methods for analyzing physical or emotional characteristics from input information and identifying the necessary components for a virtual space.

[0351] An "emotion recognition device" is a general term for hardware and software that acquires biometric information such as the user's facial expressions, voice tone, and heart rate, and analyzes the user's emotional state in real time.

[0352] "Dynamic adjustment methods" refer to methods for creating an environment that responds to the user's emotions by changing the design and elements of the virtual environment based on emotional data obtained in real time.

[0353] A "virtual space" is a three-dimensional digital environment generated by a computer, a simulation in which users can have an immersive experience through sight and interaction.

[0354] "Means of enabling two-way editing" refers to technologies that allow users to change designs and settings within a virtual space in real time, and for those changes to be immediately reflected in the system.

[0355] This invention is a system that dynamically adjusts the design and environment of a virtual space based on the user's emotions. The system consists primarily of a user, a server, and a terminal. First, the user inputs information related to the scenario and visual concept through an interface. The interface includes text input forms and image upload functions, allowing for detailed scene configuration.

[0356] The server receives information provided by the user and analyzes it using natural language processing technology. This analysis extracts environmental elements from the scenario and prepares the system for utilizing emotional data acquired in real time by the user's emotion recognition device. The emotion recognition device integrates cameras and biometric sensors to monitor the user's facial expressions, voice tone, and heart rate. This data is analyzed by an AI algorithm to identify the user's emotional state.

[0357] The server dynamically adjusts environmental elements within the virtual space based on emotional data. For example, if the server determines that the user is relaxed, it generates a design with soft colors and soothing music. The generated environment is then converted into a virtual reality format and delivered to the user in real time via the terminal.

[0358] The terminal allows users to visually experience a virtual space using devices such as VR headsets. Users can interactively edit within this virtual space, and those changes are immediately reflected in the system. For example, if a user wants to create a sad scene, the emotion engine detects that emotion, and the server adjusts the environment to a darker color scheme and changes the music to emphasize the sad atmosphere.

[0359] An example of a prompt to input into the generation AI model might be, "If the user is feeling anxious, generate a calm design that matches that emotion." This system allows users to flexibly and effectively design virtual environments and provide emotionally rich experiences.

[0360] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0361] Step 1:

[0362] Users input scenario information through an interface. The interface includes text input forms and image upload functions, allowing users to provide text and images related to movie scenes. The data obtained as input is sent to the server for analysis.

[0363] Step 2:

[0364] The server analyzes the received input data and extracts environmental elements from the scenario using natural language processing techniques. The input here is the scenario information provided by the user, and the output is a list of environmental elements necessary for initial setup of the virtual space. Specifically, it performs keyword extraction and sentiment tone analysis from the text information.

[0365] Step 3:

[0366] The emotion recognition device collects real-time emotional data from the user. Inputs include the user's facial expressions, voice tone, and heart rate. This data is processed by an emotion analysis algorithm, which outputs the user's current emotional state (e.g., joy, sadness, tension).

[0367] Step 4:

[0368] The server dynamically adjusts the virtual space design using emotion data obtained from emotion recognition. Inputs include a list of environment elements and the user's emotional state, while output is the adjusted virtual space design. Specific actions include changing the environment's color scheme, selecting music, and applying dynamic background effects.

[0369] Step 5:

[0370] The terminal converts the adjusted virtual space into VR format and provides it to the user in real time. The user can visually experience the virtual space and move freely using a VR headset. The input here is adjusted design data from the server, and the output is a visualized three-dimensional virtual environment.

[0371] Step 6:

[0372] Users can interactively edit within the virtual space and have their changes reflected instantly. User input is sent to the server as edited content, and the updated design is output. This includes actions such as rearranging objects and readjusting color tones.

[0373] (Application Example 2)

[0374] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0375] In virtual reality viewing experiences, it is difficult to dynamically adjust the environment in real time according to the user's emotional state. Therefore, there is a challenge in achieving sufficient emotional resonance between the content the user is viewing and their emotional state at that time.

[0376] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0377] In this invention, the server includes means for recognizing the user's emotional state and dynamically adjusting visual and acoustic elements within the virtual environment based on acquired emotional data; means for analyzing digital information input by the user and extracting components of the physical environment from the scenario; and means for creating a design for a three-dimensional structure that is automatically generated based on previously accumulated data. This enables real-time adjustment of the viewing environment in accordance with the user's emotions.

[0378] "Emotional state" refers to the user's current psychological tendency, identified based on physiological and psychological data such as the user's facial expressions, voice, and heart rate.

[0379] A "virtual environment" is a digital space created by a computer that provides a visual and auditory experience in a place where the user is not physically present.

[0380] "Visual elements" refer to visual content such as colors, shapes, and movements that are presented to the user within a virtual environment.

[0381] "Audio elements" refer to audio content such as sounds, music, and sound effects presented to the user within a virtual environment.

[0382] "Dynamic adjustment" refers to updating and changing the environment in real time according to the user's emotional state.

[0383] "Digital information" refers to electronic data that users provide to the system, such as movie scenes, images, and text documents.

[0384] "Components of the physical environment" refer to the visual and physical characteristics that are extracted from digital information and should be reproduced within the virtual environment.

[0385] A "three-dimensional structure" refers to an object that is generated three-dimensionally within a virtual environment and can be visually perceived by the user.

[0386] To realize this invention, a system is needed that recognizes the user's emotional state in real time and dynamically adjusts the virtual reality space accordingly. The outline of this system is described below.

[0387] The server utilizes high-performance computing devices and leverages multiple digital information processing middleware. It analyzes movie scenes and text information provided by the user through the interface using natural language processing techniques. Here, the Google Cloud Natural Language API is used to extract the physical environment's components and emotional tone from the scenario. Based on this analysis, the server uses a generative AI model to concretize the virtual environment's design and generates a three-dimensional structure using Unity.

[0388] The user's emotional state is detected by a head-mounted display and physiological sensors, and analyzed by a program utilizing the Microsoft Azure Emotion API. The acquired emotional data is sent to a server to adjust the visual and auditory elements within the virtual reality space, causing the environment to change in real time.

[0389] This system allows for adjustments to visual and auditory elements in response to the user's emotional state, such as tension or relaxation. For example, if a user is relaxed while watching a movie, the server will change the scene to warmer colors and provide soothing music. This dynamic adjustment allows the user to experience a deeper sense of immersion.

[0390] As an example of a prompt, you could enter instructions such as, "Provide emotional visual and sound changes based on a movie scene. If the user is relaxed, select a bright and relaxing soundtrack for the scenery."

[0391] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0392] Step 1:

[0393] Users input movie scenes and text information into the system via a terminal. The entered digital information is sent to the server as initial data. The server receives this data and prepares it for analysis.

[0394] Step 2:

[0395] The server uses the Google Cloud Natural Language API to analyze the digital information it receives. Specifically, it extracts elements of the physical environment and emotional tone from the scenario. It processes the scenario as input data and outputs it as "emotional tone" and "environmental elements."

[0396] Step 3:

[0397] The server uses a generative AI model to create a virtual environment design based on the analysis of emotional tone. This AI model automatically generates the optimal environment design based on past database data and user-specified prompts. The generated design data is output as three-dimensional structure data.

[0398] Step 4:

[0399] The user's emotional state is detected through a head-mounted display and physiological sensors attached to the device. The acquired emotional data is immediately analyzed using the Microsoft Azure Emotion API. The "user's emotional state" is extracted from this raw emotional data.

[0400] Step 5:

[0401] The server dynamically constructs a virtual reality environment through the Unity engine, utilizing the user's emotional state and 3D structure data. Visual and acoustic elements are adjusted in real time according to the emotional state, providing the user with a visually and acoustically optimized experience.

[0402] Step 6:

[0403] Users experience a tuned virtual environment through a head-mounted display. The visualized environment is interactive, allowing users to move freely and interact directly with it. This feedback is sent to the server in real time, and the data model is updated to inform future environment adjustments.

[0404] By coordinating each step, an interactive and real-time virtual reality environment based on the user's emotions is realized.

[0405] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0406] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0407] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0408] [Third Embodiment]

[0409] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0410] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0411] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0412] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0413] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0414] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0415] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0416] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0417] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0418] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0419] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0420] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0421] The system of this invention aims to automatically generate a three-dimensional set design for video production based on user input, and to display and edit it in a virtual reality space. The system components and their functions are described below.

[0422] The system includes an interface for users to input scenarios and visual concepts. Users can upload scripts in text format and related reference image files. For example, they can input descriptions and settings for specific scenes in a film.

[0423] The server analyzes the input digital information and extracts key elements from the scenario. This process involves using natural language processing techniques to identify information such as location, emotional tone, and theme, and then determining the design direction based on that information.

[0424] Next, the server references a database of previously accumulated data and automatically generates a 3D structure design that fits the scenario. This process utilizes AI algorithms to enhance design quality, taking style and trends into consideration. Equally important is the ability to customize the design according to user requirements.

[0425] The generated design is incorporated into a virtual reality space for viewing on a device. Using a VR headset, users can interactively experience the design and examine its details. Furthermore, users can optimize the set design by directly adjusting object placement and changing colors and textures within the virtual reality space.

[0426] As a concrete example, consider a scenario where a user wants to create a scene for a fantasy film set in a medieval castle. The user inputs details about the castle's interior and surroundings into the scenario and uploads reference images. The server analyzes this information and automatically generates structures inspired by medieval architectural styles. The user can then view the design in a VR space and adjust wall textures and furniture placement in real time as needed.

[0427] In this way, the system of the present invention allows filmmakers to significantly reduce the time and cost of set design production and improve the efficiency of the creative process.

[0428] The following describes the processing flow.

[0429] Step 1:

[0430] Users input movie scripts and visual concepts into the system through an input interface. Text data and reference images can also be uploaded.

[0431] Step 2:

[0432] The server analyzes the digital information received from the user using natural language processing technology. It extracts important elements such as location, time period, and emotional tone from the scenario text.

[0433] Step 3:

[0434] The server automatically generates a 3D structure design based on the extracted elements and by referencing a database of previously accumulated data. It utilizes AI algorithms to construct a visual concept that takes style and historical context into account.

[0435] Step 4:

[0436] The server converts the generated 3D design into a virtual reality (VR) format. It then creates a dataset for VR rendering and prepares it for transmission to the terminal.

[0437] Step 5:

[0438] The terminal receives VR data from the server, allowing users to visually experience three-dimensional designs using a VR headset. It provides a real-time preview environment.

[0439] Step 6:

[0440] Users can view and interactively edit 3D set designs within a VR space. Specifically, they can change the placement, color, and texture of objects.

[0441] Step 7:

[0442] The server receives user edits and feedback, and performs re-analysis. The data model is updated as needed to improve the next design generation process.

[0443] (Example 1)

[0444] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0445] Traditional set design for video production has faced challenges such as being too time-consuming and costly. Furthermore, the construction of physical sets lacks flexibility, making rapid design changes and customization difficult. Additionally, traditional methods fail to efficiently utilize past designs and styles, highlighting the need for improved efficiency in the creative process.

[0446] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0447] In this invention, the server includes means for analyzing user-inputted information and extracting key environmental elements from a scenario, means for creating an automatically generated spatial design based on the collected data, and means for improving the quality of the design using an artificial intelligence algorithm. This enables efficient and flexible set design creation and rapid customization in video production.

[0448] "User-inputted information" refers to scenarios, visual concepts, and related reference images provided by users of the system.

[0449] "Analysis" is the process of understanding the content and intent based on the information entered by the user, and identifying important elements.

[0450] "Key environmental elements" refer to key information that influences the design, such as settings, locations, and emotional tones extracted from the scenario.

[0451] "Collected data" refers to the collection of information about past cases, designs, and styles accumulated within the system.

[0452] "Automatically generated spatial design" refers to the design of three-dimensional structures and virtual spaces that a system automatically generates based on input information and accumulated data.

[0453] An "artificial intelligence algorithm" is a set of advanced computational methods and procedures for data analysis and design generation, possessing learning and predictive capabilities.

[0454] "Interactive editing" refers to two-way editing operations that users can perform within a virtual space, such as rearranging objects or changing their colors and textures.

[0455] An "information model" refers to the data structure and logical representation of data used within a system.

[0456] "Reflecting a modern sensibility" means incorporating elements of past designs and styles into new designs and updating them to suit contemporary trends and styles.

[0457] The embodiments for carrying out the present invention will be described in detail below.

[0458] Users utilize the system's interface to create three-dimensional set designs necessary for video production. The user interface supports uploading text-based scenarios and related images, allowing users to input specific visual concepts. For example, a user can receive assistance in generating a concrete design by entering a prompt such as, "Generate an interior design for a medieval castle. Please use the following images as reference."

[0459] The server uses natural language processing technology and artificial intelligence algorithms to process data received from the user. At this stage, it extracts environmental elements from the scenario and automatically generates a three-dimensional space based on past design data. To improve the quality of the design, the server utilizes sophisticated computing power to provide high-quality designs that meet the user's requirements.

[0460] The generated design can be displayed on the device. The device utilizes virtual reality technology to provide an environment where the user can experience the design through a VR headset. Users can further refine the design by utilizing interactive editing functions, such as adjusting the placement of objects and changing colors and textures within the VR space.

[0461] This allows users to proceed with the set design process efficiently and flexibly, significantly reducing the time and cost involved in production.

[0462] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0463] Step 1:

[0464] The user inputs the visual concept necessary for video production through the system interface. In this step, the user provides a scenario in text format and reference images. The system then receives this data and prepares it for the next processing step. Specifically, the user clicks an upload button and selects image files from their PC's storage.

[0465] Step 2:

[0466] The server analyzes information sent by the user. Based on the input scenario, it uses natural language processing techniques to extract environmental elements and emotional tones. The input includes text data, and the output generates a list of extracted keywords and concepts. This process involves calling AI algorithms to perform the processing.

[0467] Step 3:

[0468] The server references its own database and automatically generates a 3D spatial design based on the extracted information. The AI ​​algorithm creates design proposals that consider style and trends based on the input data and past design examples. As output, a high-quality 3D design that matches the theme specified by the user is generated. The server converts this into virtual reality data to pass it on to the next process.

[0469] Step 4:

[0470] The terminal places the generated 3D design in a virtual reality space, allowing the user to view the design using a VR headset. VR content is passed to the terminal as input, and a visually immersive design space is provided as output. Specifically, the VR system outputs images to the display, allowing the user to visually confirm the design.

[0471] Step 5:

[0472] Users interactively edit designs in a VR space. As input, users select objects using the headset controllers, and the output is the change in the object's position and appearance. Specifically, users can adjust the object's position by dragging and select colors and textures from menus.

[0473] (Application Example 1)

[0474] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0475] Modern content delivery services require the efficient provision of diverse and personalized virtual environments that meet user needs. Traditional methods involve significant time and cost for manual design creation, making rapid customization to meet user requirements difficult. To address this, a system is needed that can quickly and efficiently generate and edit the virtual environments users desire.

[0476] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0477] In this invention, the server includes means for analyzing digital information input by the user and extracting components of the physical environment from the scenario; means for creating a design for a three-dimensional structure that is automatically generated based on previously accumulated data; means for visually displaying the three-dimensional structure generated in the virtual reality space and enabling interactive editing; and means for dynamically generating a three-dimensional background based on a theme selected by the user and customizing the virtual environment. This allows users to quickly and efficiently generate and edit virtual spaces according to their desired themes, enabling the provision of more advanced and personalized content experiences.

[0478] A "user" is an individual or group that uses the system to input digital information and create and edit virtual environments.

[0479] "Digital information" refers to text-based data and image materials, including scenarios and visual concepts, that users input into the system.

[0480] "Analysis" is the process of processing input digital information to extract important elements and features of a scenario.

[0481] "Components of the physical environment" refers to the specific elements and attributes of the virtual reality space defined based on the scenario, and is a general term that includes buildings, terrain, and so on.

[0482] A "three-dimensional structure" is a three-dimensional object or building generated within a virtual reality space.

[0483] "Automatically generated" refers to a process that is created autonomously by a system under program control, without requiring human intervention.

[0484] "Interactive editing" refers to the ability for users to change the placement and attributes of objects in a virtual reality space in real time.

[0485] "Dynamic generation" refers to a process where content is generated while instantly changing in response to user input and circumstances.

[0486] A "virtual space" is an artificial three-dimensional space constructed by a computer, in which users can interact through an interface.

[0487] To implement this invention, three main elements are required: a server, a terminal, and a user.

[0488] server:

[0489] The server receives digital information input from the user and performs analysis using natural language processing techniques. Specifically, it uses a natural language processing library implemented in Python to extract the components of the physical environment from the scenario. Based on this information, a generative AI model operating internally using TensorFlow automatically generates appropriate three-dimensional structures. As a result, the generated three-dimensional design can be customized according to the user's requirements.

[0490] Terminal:

[0491] The device runs an application created using Unity, visually displaying three-dimensional structures generated in a virtual reality space to the user. By using an interface such as a VR headset or smart glasses, interactive editing of each object becomes possible. Users can directly move objects and change their attributes within this space, allowing for visual and intuitive design optimization.

[0492] User:

[0493] Users have the ability to input scenarios and visual concepts through the provided interface and instantly generate and edit the necessary 3D backgrounds. For example, if a user selects the "Summer Beach" theme and specifies certain elements as prompts, the system automatically creates a virtual environment that matches that theme. Through this process, users can obtain a richer virtual experience.

[0494] Specific example:

[0495] If a user wants to create a unique background for an online event, the system will generate a detailed 3D background based on the entered theme, including, for example, a coastline or a starry night sky. This generation process is done in real time, allowing users to instantly see the results and edit them as needed.

[0496] Example of a prompt:

[0497] "User-selected theme: Summer Beach. Generate a 3D background with related textures and sounds."

[0498] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0499] Step 1:

[0500] Users input scenarios and visual concepts through the system interface. This input data consists of text-based scenarios and associated image files. This data is then sent to the server as basic information to express the user's creative intent.

[0501] Step 2:

[0502] The server analyzes the received digital information using natural language processing techniques. Here, a natural language processing library implemented in Python is used to extract the components and main themes of the physical environment from text data. The input consists of text and images provided by the user, and the output is the extracted component and theme information. This output forms the basis for 3D design generation.

[0503] Step 3:

[0504] Based on the analysis results, the server references previously accumulated data and utilizes a generated AI model to design an appropriate three-dimensional structure. In this process, an initial three-dimensional model is generated based on the analyzed theme and style information. The input is the theme information and accumulated data obtained in step 2, and the output is the automatically generated three-dimensional design.

[0505] Step 4:

[0506] The 3D structure design generated by the server is sent to the terminal and placed in a virtual reality space by an application using Unity. The terminal then displays the design visually to the user through a VR headset or smart glasses. The input is the generated 3D design, and the output is the virtual space visualized by the user.

[0507] Step 5:

[0508] Users interactively edit designs within a virtual reality space. Specifically, they can adjust the position and attributes of objects in real time via their device. The virtual space obtained in the previous step is the input, and the output is the final design after the user's edits.

[0509] Step 6:

[0510] After editing is complete, the server analyzes the user's editing information and updates the system's data model. This ensures that the user's preferences and style are reflected in subsequent design generation. The input is the user's editing history, and the output is the updated data model.

[0511] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0512] The present invention provides a system that recognizes a user's emotions in real time and dynamically adjusts the design and environment of a virtual reality space based on those emotions. Specific embodiments of the present invention are described below.

[0513] The system includes an interface for users to input scenarios and visual concepts. Users can upload text information and image materials related to movie scenes, providing basic design information. For example, they can specify elements that correspond to the scene setting or the characters' psychology.

[0514] The server analyzes the input digital information and extracts the necessary components. Using natural language processing technology, it identifies elements such as location and emotional tone from the scenario and determines the direction of the generated design.

[0515] The emotion engine utilizes devices and sensors to recognize the user's emotions in real time. It records the user's facial expressions, voice tone, and heart rate, and analyzes their emotions using AI algorithms. This emotional data is then used to dynamically influence the design.

[0516] The server uses data provided by the emotion engine to modify the visual concept of the three-dimensional structure. If the user is feeling tense, the system can adjust the design to create a more relaxed atmosphere. Conversely, if a lively scene is desired, it will provide a more vibrant and energetic design.

[0517] The generated design is converted into a virtual reality format and delivered to the user in real time via the device. The user can use a VR headset to visually check the design in a three-dimensional space and interactively edit it while moving around the environment.

[0518] As a concrete example, consider a situation where a user needs to create a sad scene. When the emotion engine recognizes the user's emotions and detects sadness, the server darkens the entire environment and alters the sound effects to enhance the atmosphere. Such real-time adjustments allow filmmakers to efficiently create emotionally rich scenes.

[0519] This invention allows filmmakers to gain emotional insights into set design and construct scenes that evoke a deep emotional connection with the audience.

[0520] The following describes the processing flow.

[0521] Step 1:

[0522] Users input film scripts and visual concepts through the interface. They upload script text and related images to the system, providing foundational information to concretize the design direction.

[0523] Step 2:

[0524] The server analyzes the digital information received from the user. Using natural language processing technology, it extracts elements such as location, historical context, and emotional tone from the scenario to derive design directions.

[0525] Step 3:

[0526] The server acquires user emotional data through an emotion engine. The emotion engine processes biometric information such as the user's facial expressions, voice, and heart rate using an emotion analysis algorithm to identify their emotional state.

[0527] Step 4:

[0528] The server determines the design of the 3D structure based on extracted scenario elements and user emotion data. It dynamically adjusts the colors, lighting, and details of the environment and structure to match the user's emotions.

[0529] Step 5:

[0530] The server converts the generated design into a virtual reality format and prepares it to be delivered to the user via the terminal.

[0531] Step 6:

[0532] The device displays a three-dimensional design in a virtual reality space in real time using the user's VR headset. The user visually confirms the design in the VR environment and receives feedback that reflects their emotions.

[0533] Step 7:

[0534] Users can perform interactive editing directly within the VR space. They can optimize the design by adjusting object placement and environmental changes as needed.

[0535] Step 8:

[0536] The server receives user edits and emotional responses and updates its data model. The server then uses this information to refine future design generation.

[0537] (Example 2)

[0538] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0539] Current virtual space design systems struggle to reflect users' emotional states in real time and dynamically adjust environmental elements, creating a need for methods to provide more emotionally immersive experiences in the fields of entertainment and education. Furthermore, there is a lack of technical means to quickly and intuitively edit designs and reflect those changes throughout the entire system.

[0540] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0541] In this invention, the server includes means for analyzing user input and extracting environmental elements from a scenario; means for analyzing the user's emotional state in real time using an emotion recognition device and dynamically adjusting the environmental elements based on that analysis; and means for visually displaying the dynamically changing environment in the virtual space and enabling interactive editing. This makes it possible to generate an attractive virtual environment that reflects the user's emotions and to edit the design interactively.

[0542] "User-inputted information" refers to text and image data related to the scenario or visual concept that the user provides through the interface.

[0543] "Means for extracting environmental elements" refers to methods for analyzing physical or emotional characteristics from input information and identifying the necessary components for a virtual space.

[0544] An "emotion recognition device" is a general term for hardware and software that acquires biometric information such as the user's facial expressions, voice tone, and heart rate, and analyzes the user's emotional state in real time.

[0545] "Dynamic adjustment methods" refer to methods for creating an environment that responds to the user's emotions by changing the design and elements of the virtual environment based on emotional data obtained in real time.

[0546] A "virtual space" is a three-dimensional digital environment generated by a computer, a simulation in which users can have an immersive experience through sight and interaction.

[0547] "Means of enabling two-way editing" refers to technologies that allow users to change designs and settings within a virtual space in real time, and for those changes to be immediately reflected in the system.

[0548] This invention is a system that dynamically adjusts the design and environment of a virtual space based on the user's emotions. The system consists primarily of a user, a server, and a terminal. First, the user inputs information related to the scenario and visual concept through an interface. The interface includes text input forms and image upload functions, allowing for detailed scene configuration.

[0549] The server receives information provided by the user and analyzes it using natural language processing technology. This analysis extracts environmental elements from the scenario and prepares the system for utilizing emotional data acquired in real time by the user's emotion recognition device. The emotion recognition device integrates cameras and biometric sensors to monitor the user's facial expressions, voice tone, and heart rate. This data is analyzed by an AI algorithm to identify the user's emotional state.

[0550] The server dynamically adjusts environmental elements within the virtual space based on emotional data. For example, if the server determines that the user is relaxed, it generates a design with soft colors and soothing music. The generated environment is then converted into a virtual reality format and delivered to the user in real time via the terminal.

[0551] The terminal allows users to visually experience a virtual space using devices such as VR headsets. Users can interactively edit within this virtual space, and those changes are immediately reflected in the system. For example, if a user wants to create a sad scene, the emotion engine detects that emotion, and the server adjusts the environment to a darker color scheme and changes the music to emphasize the sad atmosphere.

[0552] An example of a prompt to input into the generation AI model might be, "If the user is feeling anxious, generate a calm design that matches that emotion." This system allows users to flexibly and effectively design virtual environments and provide emotionally rich experiences.

[0553] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0554] Step 1:

[0555] Users input scenario information through an interface. The interface includes text input forms and image upload functions, allowing users to provide text and images related to movie scenes. The data obtained as input is sent to the server for analysis.

[0556] Step 2:

[0557] The server analyzes the received input data and extracts environmental elements from the scenario using natural language processing techniques. The input here is the scenario information provided by the user, and the output is a list of environmental elements necessary for initial setup of the virtual space. Specifically, it performs keyword extraction and sentiment tone analysis from the text information.

[0558] Step 3:

[0559] The emotion recognition device collects real-time emotional data from the user. Inputs include the user's facial expressions, voice tone, and heart rate. This data is processed by an emotion analysis algorithm, which outputs the user's current emotional state (e.g., joy, sadness, tension).

[0560] Step 4:

[0561] The server dynamically adjusts the virtual space design using emotion data obtained from emotion recognition. Inputs include a list of environment elements and the user's emotional state, while output is the adjusted virtual space design. Specific actions include changing the environment's color scheme, selecting music, and applying dynamic background effects.

[0562] Step 5:

[0563] The terminal converts the adjusted virtual space into VR format and provides it to the user in real time. The user can visually experience the virtual space and move freely using a VR headset. The input here is adjusted design data from the server, and the output is a visualized three-dimensional virtual environment.

[0564] Step 6:

[0565] Users can interactively edit within the virtual space and have their changes reflected instantly. User input is sent to the server as edited content, and the updated design is output. This includes actions such as rearranging objects and readjusting color tones.

[0566] (Application Example 2)

[0567] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0568] In virtual reality viewing experiences, it is difficult to dynamically adjust the environment in real time according to the user's emotional state. Therefore, there is a challenge in achieving sufficient emotional resonance between the content the user is viewing and their emotional state at that time.

[0569] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0570] In this invention, the server includes means for recognizing the user's emotional state and dynamically adjusting visual and acoustic elements within the virtual environment based on acquired emotional data; means for analyzing digital information input by the user and extracting components of the physical environment from the scenario; and means for creating a design for a three-dimensional structure that is automatically generated based on previously accumulated data. This enables real-time adjustment of the viewing environment in accordance with the user's emotions.

[0571] "Emotional state" refers to the user's current psychological tendency, identified based on physiological and psychological data such as the user's facial expressions, voice, and heart rate.

[0572] A "virtual environment" is a digital space created by a computer that provides a visual and auditory experience in a place where the user is not physically present.

[0573] "Visual elements" refer to visual content such as colors, shapes, and movements that are presented to the user within a virtual environment.

[0574] "Audio elements" refer to audio content such as sounds, music, and sound effects presented to the user within a virtual environment.

[0575] "Dynamic adjustment" refers to updating and changing the environment in real time according to the user's emotional state.

[0576] "Digital information" refers to electronic data that users provide to the system, such as movie scenes, images, and text documents.

[0577] "Components of the physical environment" refer to the visual and physical characteristics that are extracted from digital information and should be reproduced within the virtual environment.

[0578] A "three-dimensional structure" refers to an object that is generated three-dimensionally within a virtual environment and can be visually perceived by the user.

[0579] To realize this invention, a system is needed that recognizes the user's emotional state in real time and dynamically adjusts the virtual reality space accordingly. The outline of this system is described below.

[0580] The server utilizes high-performance computing devices and leverages multiple digital information processing middleware. It analyzes movie scenes and text information provided by the user through the interface using natural language processing techniques. Here, the Google Cloud Natural Language API is used to extract the physical environment's components and emotional tone from the scenario. Based on this analysis, the server uses a generative AI model to concretize the virtual environment's design and generates a three-dimensional structure using Unity.

[0581] The user's emotional state is detected by a head-mounted display and physiological sensors, and analyzed by a program utilizing the Microsoft Azure Emotion API. The acquired emotional data is sent to a server to adjust the visual and auditory elements within the virtual reality space, causing the environment to change in real time.

[0582] This system allows for adjustments to visual and auditory elements in response to the user's emotional state, such as tension or relaxation. For example, if a user is relaxed while watching a movie, the server will change the scene to warmer colors and provide soothing music. This dynamic adjustment allows the user to experience a deeper sense of immersion.

[0583] As an example of a prompt, you could enter instructions such as, "Provide emotional visual and sound changes based on a movie scene. If the user is relaxed, select a bright and relaxing soundtrack for the scenery."

[0584] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0585] Step 1:

[0586] Users input movie scenes and text information into the system via a terminal. The entered digital information is sent to the server as initial data. The server receives this data and prepares it for analysis.

[0587] Step 2:

[0588] The server uses the Google Cloud Natural Language API to analyze the digital information it receives. Specifically, it extracts elements of the physical environment and emotional tone from the scenario. It processes the scenario as input data and outputs it as "emotional tone" and "environmental elements."

[0589] Step 3:

[0590] The server uses a generative AI model to create a virtual environment design based on the analysis of emotional tone. This AI model automatically generates the optimal environment design based on past database data and user-specified prompts. The generated design data is output as three-dimensional structure data.

[0591] Step 4:

[0592] The user's emotional state is detected through a head-mounted display and physiological sensors attached to the device. The acquired emotional data is immediately analyzed using the Microsoft Azure Emotion API. The "user's emotional state" is extracted from this raw emotional data.

[0593] Step 5:

[0594] The server dynamically constructs a virtual reality environment through the Unity engine, utilizing the user's emotional state and 3D structure data. Visual and acoustic elements are adjusted in real time according to the emotional state, providing the user with a visually and acoustically optimized experience.

[0595] Step 6:

[0596] Users experience a tuned virtual environment through a head-mounted display. The visualized environment is interactive, allowing users to move freely and interact directly with it. This feedback is sent to the server in real time, and the data model is updated to inform future environment adjustments.

[0597] By coordinating each step, an interactive and real-time virtual reality environment based on the user's emotions is realized.

[0598] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0599] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0600] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0601] [Fourth Embodiment]

[0602] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0603] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0604] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0605] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0606] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0607] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0608] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0609] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0610] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0611] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0612] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0613] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0614] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0615] The system of this invention aims to automatically generate a three-dimensional set design for video production based on user input, and to display and edit it in a virtual reality space. The system components and their functions are described below.

[0616] The system includes an interface for users to input scenarios and visual concepts. Users can upload scripts in text format and related reference image files. For example, they can input descriptions and settings for specific scenes in a film.

[0617] The server analyzes the input digital information and extracts key elements from the scenario. This process involves using natural language processing techniques to identify information such as location, emotional tone, and theme, and then determining the design direction based on that information.

[0618] Next, the server references a database of previously accumulated data and automatically generates a 3D structure design that fits the scenario. This process utilizes AI algorithms to enhance design quality, taking style and trends into consideration. Equally important is the ability to customize the design according to user requirements.

[0619] The generated design is incorporated into a virtual reality space for viewing on a device. Using a VR headset, users can interactively experience the design and examine its details. Furthermore, users can optimize the set design by directly adjusting object placement and changing colors and textures within the virtual reality space.

[0620] As a concrete example, consider a scenario where a user wants to create a scene for a fantasy film set in a medieval castle. The user inputs details about the castle's interior and surroundings into the scenario and uploads reference images. The server analyzes this information and automatically generates structures inspired by medieval architectural styles. The user can then view the design in a VR space and adjust wall textures and furniture placement in real time as needed.

[0621] In this way, the system of the present invention allows filmmakers to significantly reduce the time and cost of set design production and improve the efficiency of the creative process.

[0622] The following describes the processing flow.

[0623] Step 1:

[0624] Users input movie scripts and visual concepts into the system through an input interface. Text data and reference images can also be uploaded.

[0625] Step 2:

[0626] The server analyzes the digital information received from the user using natural language processing technology. It extracts important elements such as location, time period, and emotional tone from the scenario text.

[0627] Step 3:

[0628] The server automatically generates a 3D structure design based on the extracted elements and by referencing a database of previously accumulated data. It utilizes AI algorithms to construct a visual concept that takes style and historical context into account.

[0629] Step 4:

[0630] The server converts the generated 3D design into a virtual reality (VR) format. It then creates a dataset for VR rendering and prepares it for transmission to the terminal.

[0631] Step 5:

[0632] The terminal receives VR data from the server, allowing users to visually experience three-dimensional designs using a VR headset. It provides a real-time preview environment.

[0633] Step 6:

[0634] Users can view and interactively edit 3D set designs within a VR space. Specifically, they can change the placement, color, and texture of objects.

[0635] Step 7:

[0636] The server receives user edits and feedback, and performs re-analysis. The data model is updated as needed to improve the next design generation process.

[0637] (Example 1)

[0638] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0639] Traditional set design for video production has faced challenges such as being too time-consuming and costly. Furthermore, the construction of physical sets lacks flexibility, making rapid design changes and customization difficult. Additionally, traditional methods fail to efficiently utilize past designs and styles, highlighting the need for improved efficiency in the creative process.

[0640] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0641] In this invention, the server includes means for analyzing user-inputted information and extracting key environmental elements from a scenario, means for creating an automatically generated spatial design based on the collected data, and means for improving the quality of the design using an artificial intelligence algorithm. This enables efficient and flexible set design creation and rapid customization in video production.

[0642] "User-inputted information" refers to scenarios, visual concepts, and related reference images provided by users of the system.

[0643] "Analysis" is the process of understanding the content and intent based on the information entered by the user, and identifying important elements.

[0644] "Key environmental elements" refer to key information that influences the design, such as settings, locations, and emotional tones extracted from the scenario.

[0645] "Collected data" refers to the collection of information about past cases, designs, and styles accumulated within the system.

[0646] "Automatically generated spatial design" refers to the design of three-dimensional structures and virtual spaces that a system automatically generates based on input information and accumulated data.

[0647] An "artificial intelligence algorithm" is a set of advanced computational methods and procedures for data analysis and design generation, possessing learning and predictive capabilities.

[0648] "Interactive editing" refers to two-way editing operations that users can perform within a virtual space, such as rearranging objects or changing their colors and textures.

[0649] An "information model" refers to the data structure and logical representation of data used within a system.

[0650] "Reflecting a modern sensibility" means incorporating elements of past designs and styles into new designs and updating them to suit contemporary trends and styles.

[0651] The embodiments for carrying out the present invention will be described in detail below.

[0652] Users utilize the system's interface to create three-dimensional set designs necessary for video production. The user interface supports uploading text-based scenarios and related images, allowing users to input specific visual concepts. For example, a user can receive assistance in generating a concrete design by entering a prompt such as, "Generate an interior design for a medieval castle. Please use the following images as reference."

[0653] The server uses natural language processing technology and artificial intelligence algorithms to process data received from the user. At this stage, it extracts environmental elements from the scenario and automatically generates a three-dimensional space based on past design data. To improve the quality of the design, the server utilizes sophisticated computing power to provide high-quality designs that meet the user's requirements.

[0654] The generated design can be displayed on the device. The device utilizes virtual reality technology to provide an environment where the user can experience the design through a VR headset. Users can further refine the design by utilizing interactive editing functions, such as adjusting the placement of objects and changing colors and textures within the VR space.

[0655] This allows users to proceed with the set design process efficiently and flexibly, significantly reducing the time and cost involved in production.

[0656] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0657] Step 1:

[0658] The user inputs the visual concept necessary for video production through the system interface. In this step, the user provides a scenario in text format and reference images. The system then receives this data and prepares it for the next processing step. Specifically, the user clicks an upload button and selects image files from their PC's storage.

[0659] Step 2:

[0660] The server analyzes information sent by the user. Based on the input scenario, it uses natural language processing techniques to extract environmental elements and emotional tones. The input includes text data, and the output generates a list of extracted keywords and concepts. This process involves calling AI algorithms to perform the processing.

[0661] Step 3:

[0662] The server references its own database and automatically generates a 3D spatial design based on the extracted information. The AI ​​algorithm creates design proposals that consider style and trends based on the input data and past design examples. As output, a high-quality 3D design that matches the theme specified by the user is generated. The server converts this into virtual reality data to pass it on to the next process.

[0663] Step 4:

[0664] The terminal places the generated 3D design in a virtual reality space, allowing the user to view the design using a VR headset. VR content is passed to the terminal as input, and a visually immersive design space is provided as output. Specifically, the VR system outputs images to the display, allowing the user to visually confirm the design.

[0665] Step 5:

[0666] Users interactively edit designs in a VR space. As input, users select objects using the headset controllers, and the output is the change in the object's position and appearance. Specifically, users can adjust the object's position by dragging and select colors and textures from menus.

[0667] (Application Example 1)

[0668] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0669] Modern content delivery services require the efficient provision of diverse and personalized virtual environments that meet user needs. Traditional methods involve significant time and cost for manual design creation, making rapid customization to meet user requirements difficult. To address this, a system is needed that can quickly and efficiently generate and edit the virtual environments users desire.

[0670] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0671] In this invention, the server includes means for analyzing digital information input by the user and extracting components of the physical environment from the scenario; means for creating a design for a three-dimensional structure that is automatically generated based on previously accumulated data; means for visually displaying the three-dimensional structure generated in the virtual reality space and enabling interactive editing; and means for dynamically generating a three-dimensional background based on a theme selected by the user and customizing the virtual environment. This allows users to quickly and efficiently generate and edit virtual spaces according to their desired themes, enabling the provision of more advanced and personalized content experiences.

[0672] A "user" is an individual or group that uses the system to input digital information and create and edit virtual environments.

[0673] "Digital information" refers to text-based data and image materials, including scenarios and visual concepts, that users input into the system.

[0674] "Analysis" is the process of processing input digital information to extract important elements and features of a scenario.

[0675] "Components of the physical environment" refers to the specific elements and attributes of the virtual reality space defined based on the scenario, and is a general term that includes buildings, terrain, and so on.

[0676] A "three-dimensional structure" is a three-dimensional object or building generated within a virtual reality space.

[0677] "Automatically generated" refers to a process that is created autonomously by a system under program control, without requiring human intervention.

[0678] "Interactive editing" refers to the ability for users to change the placement and attributes of objects in a virtual reality space in real time.

[0679] "Dynamic generation" refers to a process where content is generated while instantly changing in response to user input and circumstances.

[0680] A "virtual space" is an artificial three-dimensional space constructed by a computer, in which users can interact through an interface.

[0681] To implement this invention, three main elements are required: a server, a terminal, and a user.

[0682] server:

[0683] The server receives digital information input from the user and performs analysis using natural language processing techniques. Specifically, it uses a natural language processing library implemented in Python to extract the components of the physical environment from the scenario. Based on this information, a generative AI model operating internally using TensorFlow automatically generates appropriate three-dimensional structures. As a result, the generated three-dimensional design can be customized according to the user's requirements.

[0684] Terminal:

[0685] The device runs an application created using Unity, visually displaying three-dimensional structures generated in a virtual reality space to the user. By using an interface such as a VR headset or smart glasses, interactive editing of each object becomes possible. Users can directly move objects and change their attributes within this space, allowing for visual and intuitive design optimization.

[0686] User:

[0687] Users have the ability to input scenarios and visual concepts through the provided interface and instantly generate and edit the necessary 3D backgrounds. For example, if a user selects the "Summer Beach" theme and specifies certain elements as prompts, the system automatically creates a virtual environment that matches that theme. Through this process, users can obtain a richer virtual experience.

[0688] Specific example:

[0689] If a user wants to create a unique background for an online event, the system will generate a detailed 3D background based on the entered theme, including, for example, a coastline or a starry night sky. This generation process is done in real time, allowing users to instantly see the results and edit them as needed.

[0690] Example of a prompt:

[0691] "User-selected theme: Summer Beach. Generate a 3D background with related textures and sounds."

[0692] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0693] Step 1:

[0694] Users input scenarios and visual concepts through the system interface. This input data consists of text-based scenarios and associated image files. This data is then sent to the server as basic information to express the user's creative intent.

[0695] Step 2:

[0696] The server analyzes the received digital information using natural language processing techniques. Here, a natural language processing library implemented in Python is used to extract the components and main themes of the physical environment from text data. The input consists of text and images provided by the user, and the output is the extracted component and theme information. This output forms the basis for 3D design generation.

[0697] Step 3:

[0698] Based on the analysis results, the server references previously accumulated data and utilizes a generated AI model to design an appropriate three-dimensional structure. In this process, an initial three-dimensional model is generated based on the analyzed theme and style information. The input is the theme information and accumulated data obtained in step 2, and the output is the automatically generated three-dimensional design.

[0699] Step 4:

[0700] The 3D structure design generated by the server is sent to the terminal and placed in a virtual reality space by an application using Unity. The terminal then displays the design visually to the user through a VR headset or smart glasses. The input is the generated 3D design, and the output is the virtual space visualized by the user.

[0701] Step 5:

[0702] Users interactively edit designs within a virtual reality space. Specifically, they can adjust the position and attributes of objects in real time via their device. The virtual space obtained in the previous step is the input, and the output is the final design after the user's edits.

[0703] Step 6:

[0704] After editing is complete, the server analyzes the user's editing information and updates the system's data model. This ensures that the user's preferences and style are reflected in subsequent design generation. The input is the user's editing history, and the output is the updated data model.

[0705] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0706] The present invention provides a system that recognizes a user's emotions in real time and dynamically adjusts the design and environment of a virtual reality space based on those emotions. Specific embodiments of the present invention are described below.

[0707] The system includes an interface for users to input scenarios and visual concepts. Users can upload text information and image materials related to movie scenes, providing basic design information. For example, they can specify elements that correspond to the scene setting or the characters' psychology.

[0708] The server analyzes the input digital information and extracts the necessary components. Using natural language processing technology, it identifies elements such as location and emotional tone from the scenario and determines the direction of the generated design.

[0709] The emotion engine utilizes devices and sensors to recognize the user's emotions in real time. It records the user's facial expressions, voice tone, and heart rate, and analyzes their emotions using AI algorithms. This emotional data is then used to dynamically influence the design.

[0710] The server uses data provided by the emotion engine to modify the visual concept of the three-dimensional structure. If the user is feeling tense, the system can adjust the design to create a more relaxed atmosphere. Conversely, if a lively scene is desired, it will provide a more vibrant and energetic design.

[0711] The generated design is converted into a virtual reality format and delivered to the user in real time via the device. The user can use a VR headset to visually check the design in a three-dimensional space and interactively edit it while moving around the environment.

[0712] As a concrete example, consider a situation where a user needs to create a sad scene. When the emotion engine recognizes the user's emotions and detects sadness, the server darkens the entire environment and alters the sound effects to enhance the atmosphere. Such real-time adjustments allow filmmakers to efficiently create emotionally rich scenes.

[0713] This invention allows filmmakers to gain emotional insights into set design and construct scenes that evoke a deep emotional connection with the audience.

[0714] The following describes the processing flow.

[0715] Step 1:

[0716] Users input film scripts and visual concepts through the interface. They upload script text and related images to the system, providing foundational information to concretize the design direction.

[0717] Step 2:

[0718] The server analyzes the digital information received from the user. Using natural language processing technology, it extracts elements such as location, historical context, and emotional tone from the scenario to derive design directions.

[0719] Step 3:

[0720] The server acquires user emotional data through an emotion engine. The emotion engine processes biometric information such as the user's facial expressions, voice, and heart rate using an emotion analysis algorithm to identify their emotional state.

[0721] Step 4:

[0722] The server determines the design of the 3D structure based on extracted scenario elements and user emotion data. It dynamically adjusts the colors, lighting, and details of the environment and structure to match the user's emotions.

[0723] Step 5:

[0724] The server converts the generated design into a virtual reality format and prepares it to be delivered to the user via the terminal.

[0725] Step 6:

[0726] The device displays a three-dimensional design in a virtual reality space in real time using the user's VR headset. The user visually confirms the design in the VR environment and receives feedback that reflects their emotions.

[0727] Step 7:

[0728] Users can perform interactive editing directly within the VR space. They can optimize the design by adjusting object placement and environmental changes as needed.

[0729] Step 8:

[0730] The server receives user edits and emotional responses and updates its data model. The server then uses this information to refine future design generation.

[0731] (Example 2)

[0732] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0733] Current virtual space design systems struggle to reflect users' emotional states in real time and dynamically adjust environmental elements, creating a need for methods to provide more emotionally immersive experiences in the fields of entertainment and education. Furthermore, there is a lack of technical means to quickly and intuitively edit designs and reflect those changes throughout the entire system.

[0734] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0735] In this invention, the server includes means for analyzing user input and extracting environmental elements from a scenario; means for analyzing the user's emotional state in real time using an emotion recognition device and dynamically adjusting the environmental elements based on that analysis; and means for visually displaying the dynamically changing environment in the virtual space and enabling interactive editing. This makes it possible to generate an attractive virtual environment that reflects the user's emotions and to edit the design interactively.

[0736] "User-inputted information" refers to text and image data related to the scenario or visual concept that the user provides through the interface.

[0737] "Means for extracting environmental elements" refers to methods for analyzing physical or emotional characteristics from input information and identifying the necessary components for a virtual space.

[0738] An "emotion recognition device" is a general term for hardware and software that acquires biometric information such as the user's facial expressions, voice tone, and heart rate, and analyzes the user's emotional state in real time.

[0739] "Dynamic adjustment methods" refer to methods for creating an environment that responds to the user's emotions by changing the design and elements of the virtual environment based on emotional data obtained in real time.

[0740] A "virtual space" is a three-dimensional digital environment generated by a computer, a simulation in which users can have an immersive experience through sight and interaction.

[0741] "Means of enabling two-way editing" refers to technologies that allow users to change designs and settings within a virtual space in real time, and for those changes to be immediately reflected in the system.

[0742] This invention is a system that dynamically adjusts the design and environment of a virtual space based on the user's emotions. The system consists primarily of a user, a server, and a terminal. First, the user inputs information related to the scenario and visual concept through an interface. The interface includes text input forms and image upload functions, allowing for detailed scene configuration.

[0743] The server receives information provided by the user and analyzes it using natural language processing technology. This analysis extracts environmental elements from the scenario and prepares the system for utilizing emotional data acquired in real time by the user's emotion recognition device. The emotion recognition device integrates cameras and biometric sensors to monitor the user's facial expressions, voice tone, and heart rate. This data is analyzed by an AI algorithm to identify the user's emotional state.

[0744] The server dynamically adjusts environmental elements within the virtual space based on emotional data. For example, if the server determines that the user is relaxed, it generates a design with soft colors and soothing music. The generated environment is then converted into a virtual reality format and delivered to the user in real time via the terminal.

[0745] The terminal allows users to visually experience a virtual space using devices such as VR headsets. Users can interactively edit within this virtual space, and those changes are immediately reflected in the system. For example, if a user wants to create a sad scene, the emotion engine detects that emotion, and the server adjusts the environment to a darker color scheme and changes the music to emphasize the sad atmosphere.

[0746] An example of a prompt to input into the generation AI model might be, "If the user is feeling anxious, generate a calm design that matches that emotion." This system allows users to flexibly and effectively design virtual environments and provide emotionally rich experiences.

[0747] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0748] Step 1:

[0749] Users input scenario information through an interface. The interface includes text input forms and image upload functions, allowing users to provide text and images related to movie scenes. The data obtained as input is sent to the server for analysis.

[0750] Step 2:

[0751] The server analyzes the received input data and extracts environmental elements from the scenario using natural language processing techniques. The input here is the scenario information provided by the user, and the output is a list of environmental elements necessary for initial setup of the virtual space. Specifically, it performs keyword extraction and sentiment tone analysis from the text information.

[0752] Step 3:

[0753] The emotion recognition device collects real-time emotional data from the user. Inputs include the user's facial expressions, voice tone, and heart rate. This data is processed by an emotion analysis algorithm, which outputs the user's current emotional state (e.g., joy, sadness, tension).

[0754] Step 4:

[0755] The server dynamically adjusts the virtual space design using emotion data obtained from emotion recognition. Inputs include a list of environment elements and the user's emotional state, while output is the adjusted virtual space design. Specific actions include changing the environment's color scheme, selecting music, and applying dynamic background effects.

[0756] Step 5:

[0757] The terminal converts the adjusted virtual space into VR format and provides it to the user in real time. The user can visually experience the virtual space and move freely using a VR headset. The input here is adjusted design data from the server, and the output is a visualized three-dimensional virtual environment.

[0758] Step 6:

[0759] Users can interactively edit within the virtual space and have their changes reflected instantly. User input is sent to the server as edited content, and the updated design is output. This includes actions such as rearranging objects and readjusting color tones.

[0760] (Application Example 2)

[0761] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0762] In virtual reality viewing experiences, it is difficult to dynamically adjust the environment in real time according to the user's emotional state. Therefore, there is a challenge in achieving sufficient emotional resonance between the content the user is viewing and their emotional state at that time.

[0763] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0764] In this invention, the server includes means for recognizing the user's emotional state and dynamically adjusting visual and acoustic elements within the virtual environment based on acquired emotional data; means for analyzing digital information input by the user and extracting components of the physical environment from the scenario; and means for creating a design for a three-dimensional structure that is automatically generated based on previously accumulated data. This enables real-time adjustment of the viewing environment in accordance with the user's emotions.

[0765] "Emotional state" refers to the user's current psychological tendency, identified based on physiological and psychological data such as the user's facial expressions, voice, and heart rate.

[0766] A "virtual environment" is a digital space created by a computer that provides a visual and auditory experience in a place where the user is not physically present.

[0767] "Visual elements" refer to visual content such as colors, shapes, and movements that are presented to the user within a virtual environment.

[0768] "Audio elements" refer to audio content such as sounds, music, and sound effects presented to the user within a virtual environment.

[0769] "Dynamic adjustment" refers to updating and changing the environment in real time according to the user's emotional state.

[0770] "Digital information" refers to electronic data that users provide to the system, such as movie scenes, images, and text documents.

[0771] "Components of the physical environment" refer to the visual and physical characteristics that are extracted from digital information and should be reproduced within the virtual environment.

[0772] A "three-dimensional structure" refers to an object that is generated three-dimensionally within a virtual environment and can be visually perceived by the user.

[0773] To realize this invention, a system is needed that recognizes the user's emotional state in real time and dynamically adjusts the virtual reality space accordingly. The outline of this system is described below.

[0774] The server utilizes high-performance computing devices and leverages multiple digital information processing middleware. It analyzes movie scenes and text information provided by the user through the interface using natural language processing techniques. Here, the Google Cloud Natural Language API is used to extract the physical environment's components and emotional tone from the scenario. Based on this analysis, the server uses a generative AI model to concretize the virtual environment's design and generates a three-dimensional structure using Unity.

[0775] The user's emotional state is detected by a head-mounted display and physiological sensors, and analyzed by a program utilizing the Microsoft Azure Emotion API. The acquired emotional data is sent to a server to adjust the visual and auditory elements within the virtual reality space, causing the environment to change in real time.

[0776] This system allows for adjustments to visual and auditory elements in response to the user's emotional state, such as tension or relaxation. For example, if a user is relaxed while watching a movie, the server will change the scene to warmer colors and provide soothing music. This dynamic adjustment allows the user to experience a deeper sense of immersion.

[0777] As an example of a prompt, you could enter instructions such as, "Provide emotional visual and sound changes based on a movie scene. If the user is relaxed, select a bright and relaxing soundtrack for the scenery."

[0778] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0779] Step 1:

[0780] Users input movie scenes and text information into the system via a terminal. The entered digital information is sent to the server as initial data. The server receives this data and prepares it for analysis.

[0781] Step 2:

[0782] The server uses the Google Cloud Natural Language API to analyze the digital information it receives. Specifically, it extracts elements of the physical environment and emotional tone from the scenario. It processes the scenario as input data and outputs it as "emotional tone" and "environmental elements."

[0783] Step 3:

[0784] The server uses a generative AI model to create a virtual environment design based on the analysis of emotional tone. This AI model automatically generates the optimal environment design based on past database data and user-specified prompts. The generated design data is output as three-dimensional structure data.

[0785] Step 4:

[0786] The user's emotional state is detected through a head-mounted display and physiological sensors attached to the device. The acquired emotional data is immediately analyzed using the Microsoft Azure Emotion API. The "user's emotional state" is extracted from this raw emotional data.

[0787] Step 5:

[0788] The server dynamically constructs a virtual reality environment through the Unity engine, utilizing the user's emotional state and 3D structure data. Visual and acoustic elements are adjusted in real time according to the emotional state, providing the user with a visually and acoustically optimized experience.

[0789] Step 6:

[0790] Users experience a tuned virtual environment through a head-mounted display. The visualized environment is interactive, allowing users to move freely and interact directly with it. This feedback is sent to the server in real time, and the data model is updated to inform future environment adjustments.

[0791] By coordinating each step, an interactive and real-time virtual reality environment based on the user's emotions is realized.

[0792] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0793] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0794] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0795] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0796] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0797] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0798] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0799] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0800] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0801] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0802] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0803] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0804] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0805] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0806] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0807] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0808] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0809] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0810] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0811] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0812] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0813] The following is further disclosed regarding the embodiments described above.

[0814] (Claim 1)

[0815] A means for analyzing digital information entered by the user and extracting components of the physical environment from the scenario,

[0816] A means of creating a design for a three-dimensional structure that is automatically generated based on data accumulated in the past,

[0817] A means for visually displaying three-dimensional structures generated in a virtual reality space and enabling interactive editing,

[0818] A system that includes this.

[0819] (Claim 2)

[0820] The system according to claim 1, comprising means for analyzing editing information performed by a user in a virtual reality space and updating the system's data model.

[0821] (Claim 3)

[0822] The system according to claim 1, comprising means for reflecting a modern design sensibility in the generated three-dimensional structure using past visual styles in a database.

[0823] "Example 1"

[0824] (Claim 1)

[0825] A means of analyzing user input and extracting key environmental elements from the scenario,

[0826] A means of creating a spatial design that is automatically generated based on the collected data,

[0827] Methods for improving design quality using artificial intelligence algorithms,

[0828] A means of visually displaying a space generated within a virtual space and enabling interactive editing,

[0829] A system that includes this.

[0830] (Claim 2)

[0831] The system according to claim 1, comprising means for analyzing editing information performed by a user in a virtual space and updating an information model.

[0832] (Claim 3)

[0833] The system according to claim 1, comprising means of referencing past design data and reflecting a modern sensibility in the generated space.

[0834] "Application Example 1"

[0835] (Claim 1)

[0836] A means for analyzing digital information entered by the user and extracting components of the physical environment from the scenario,

[0837] A means of creating a design for a three-dimensional structure that is automatically generated based on data accumulated in the past,

[0838] A means for visually displaying three-dimensional structures generated in a virtual reality space and enabling interactive editing,

[0839] A means of dynamically generating a three-dimensional background based on a theme selected by the user and customizing the virtual environment,

[0840] A system that includes this.

[0841] (Claim 2)

[0842] The system according to claim 1, comprising means for analyzing editing information performed by a user in a virtual reality space and updating the system's data model.

[0843] (Claim 3)

[0844] The system according to claim 1, comprising means for reflecting a modern design sensibility in the generated three-dimensional structure using past visual styles in a database.

[0845] "Example 2 of combining an emotion engine"

[0846] (Claim 1)

[0847] A means of analyzing user input and extracting environmental elements from the scenario,

[0848] A means for analyzing the user's emotional state in real time using an emotion recognition device and dynamically adjusting environmental elements based on that analysis,

[0849] A means to visually display a dynamically changing environment within a virtual space and enable interactive editing,

[0850] A system that includes this.

[0851] (Claim 2)

[0852] The system according to claim 1, comprising means for analyzing editing information performed by a user in a virtual space and updating the information set.

[0853] (Claim 3)

[0854] The system according to claim 1, comprising means for reflecting a modern design sensibility in generated environmental elements by utilizing past visual styles in a memory area.

[0855] "Application example 2 when combining with an emotional engine"

[0856] (Claim 1)

[0857] A means for recognizing the user's emotional state and dynamically adjusting visual and auditory elements within the virtual environment based on acquired emotional data,

[0858] A means for analyzing digital information entered by the user and extracting components of the physical environment from the scenario,

[0859] A means of creating a design for a three-dimensional structure that is automatically generated based on data accumulated in the past,

[0860] A means for visually displaying three-dimensional structures generated in a virtual reality space and enabling interactive editing,

[0861] A system that includes this.

[0862] (Claim 2)

[0863] The system according to claim 1, comprising means for analyzing editing information and emotional data performed by a user in a virtual reality space and updating the system's data model.

[0864] (Claim 3)

[0865] The system according to claim 1, comprising means for using past visual styles in a database to reflect a modern design sensibility in the generated three-dimensional structure and dynamically changing it according to the user's emotional state. [Explanation of symbols]

[0866] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for analyzing digital information entered by the user and extracting components of the physical environment from the scenario, A means of creating a design for a three-dimensional structure that is automatically generated based on data accumulated in the past, A means for visually displaying three-dimensional structures generated in a virtual reality space and enabling interactive editing, A system that includes this.

2. The system according to claim 1, comprising means for analyzing editing information performed by a user in a virtual reality space and updating the system's data model.

3. The system according to claim 1, comprising means for reflecting a modern design sensibility in the generated three-dimensional structure using past visual styles in a database.

Images