System

Abstract

A system is provided.SOLUTION: - A system comprising: means for acquiring a designated image; means for dividing the image into a grid; means for generating and projecting the image to fill gaps between the divided areas of the grid; means for allowing a user to select a specific scene; means for converting the image into a 360-degree full screen or a rectangular display according to the selected scene; means for integrating and storing a final image for spherical AI; and means for projecting the image onto a spherical screen.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, new imaging technologies such as virtual reality (VR) and augmented reality (AR) have evolved, and consumers' expectations for visual experiences are rising. However, projecting images onto spherical screens like those used in planetariums poses the challenge of requiring highly advanced processing technology to minimize image distortion and provide a highly immersive experience. This makes it difficult to convert existing films for spherical projection, limiting opportunities for consumers to enjoy classic movies from a new perspective. This invention aims to solve this problem by developing a technology that uses generative AI to divide images into a grid and easily convert them for spherical projection while correcting for distortion. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means. First, specified video data is acquired and divided into a grid. Next, gaps between each grid-like division are filled using a generative AI means. This smooths the gaps between the meshes, enabling the video to be projected onto a sphere with high accuracy. The present invention also provides a means for a user to select a specific scene and display the video in a 360-degree full screen or as a central rectangular display depending on the selected scene. Furthermore, the present invention provides a system that includes a means for integrating and saving the final spherical projection video and projecting the video onto a spherical screen. This procedure makes it possible to convert existing movies for spherical projection, providing consumers with a new visual experience.

[0006] "Video data" means digital files of electronically recorded video or images intended to visually convey specific information.

[0007] "Mesh division" is the process of dividing an image or video frame into small units in a grid pattern.

[0008] "Generative AI methods" are algorithms or models that use artificial intelligence to generate new pixel information to fill in missing parts or gaps.

[0009] "Gap filling" is a process that fills gaps that occur between images or video frames that are divided into a mesh using information from neighboring pixels.

[0010] The "user selection means" refers to a method by which the user selects a particular scene or display format through the operation interface.

[0011] "360-degree full screen" is a display format in which an image is projected across the entire surface of a sphere (dome-shaped screen).

[0012] "Rectangular display" is a display format in which an image is projected in a rectangular shape at the center of a spherical screen.

[0013] "Video merging" is the process of combining separately processed frames or segments into one continuous video file.

[0014] A "spherical screen" is a screen for projecting images onto a dome-shaped surface. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a system that uses generative AI to convert existing video data into images that can be projected onto a 360-degree spherical screen. This system is composed of a server, terminals, and users working together, and each step plays a different role.

[0037] Server Roles

[0038] Acquiring video data

[0039] The server retrieves the specified video data file. For example, if a video file named movie.mp4 is specified, the server reads it and starts processing.

[0040] Mesh division of video

[0041] The server divides the acquired video data into a grid. For example, it divides a video frame into a 10x10 mesh and saves each mesh in a format that can be processed individually.

[0042] Interpolation between meshes

[0043] The server uses generation AI to fill in the gaps between the divided meshes. The generation AI analyzes the boundaries of each mesh and generates a complementary image based on information from surrounding pixels. This results in a smooth transition between images, making it suitable for spherical projection.

[0044] Scene selection and processing

[0045] Through the interface, the user selects whether to display a particular scene (e.g., ocean or space) in full 360-degree screen or in a central rectangular view. This selection information is sent to the server, which then performs the appropriate processing for each scene.

[0046] Merging and saving the final footage

[0047] The server merges all the processed frames and generates the final spherical projection video file, which is saved on the server in the format converted_movie.mp4, for example.

[0048] Device Role

[0049] Image projection

[0050] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. Using a projector connected to the terminal, the video is expanded and displayed on a 360-degree spherical screen, allowing the audience to experience the video in all directions.

[0051] User Roles

[0052] Select a scene

[0053] The user selects a specific scene from the video through an interface (e.g., dedicated software or a web application). Based on this selection, the system determines whether that scene should be displayed as a full 360-degree screen, while the other scenes should be displayed as a rectangular area in the center.

[0054] Viewing

[0055] Users can view images projected onto a spherical screen in the planetarium. The immersive feeling provided by 360-degree images allows users to enjoy a new visual experience that is different from that of a conventional flat screen.

[0056] Specific examples

[0057] For example, if a user wants to view an "ocean" scene in full 360-degree screen, the user selects the "ocean" scene through the interface. Based on this information, the server uses AI to generate the "ocean" scene for full 360-degree screen viewing, while other scenes (e.g., indoor scenes) are displayed as rectangular images in the center. Finally, the generated spherical projection video file is sent from the server to the device and projected onto the spherical screen of the planetarium. This process allows the user to enjoy a new visual experience.

[0058] Through these processing steps, the system of the present invention can easily convert existing images for spherical projection, providing a highly immersive experience.

[0059] The processing flow will be explained below.

[0060] Step 1: Acquire video data

[0061] The server retrieves the specified video data file. For example, if the file name is movie.mp4, the server reads this file and starts processing it. The video file is split into frames using a video processing library, and each frame is saved in a buffer.

[0062] Step 2: Meshing the image

[0063] The server divides each loaded frame into a grid. Specifically, it divides the frame into 10x10 meshes and treats each mesh as individual data. This process is performed using an image processing library.

[0064] Step 3: Interpolation between meshes

[0065] The server uses generative AI to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates pixels to fill in the gaps based on information from surrounding pixels. This filling process smooths the connections between the meshes, generating a continuous image suitable for spherical projection.

[0066] Step 4: Select a Scene

[0067] The user operates an interface to select specific scenes in the video, for example, using dedicated software or a web application. For each scene, the user can choose between a full-screen 360-degree view or a central rectangular view.

[0068] Step 5: Decide on the scene display format

[0069] The server receives the scene information selected by the user and determines the display format for each scene based on that information: certain scenes are processed for a full 360-degree screen display, while others are processed for a central rectangular display.

[0070] Step 6: Merging and saving the final footage

[0071] The server then combines all the processed frames to generate the final spherical projection video file. The resulting video file is saved on the server in the format converted_movie.mp4, for example. To do this, a video processing library is used to connect the frames and output them as a single continuous video file.

[0072] Step 7: Projecting the image

[0073] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. Using a projector connected to the terminal, the video is enlarged and displayed on a 360-degree spherical screen. This projection allows the audience to enjoy the video in all directions.

[0074] Step 8: Watch the video

[0075] Users can view images projected onto a spherical screen in the planetarium. The 360-degree images provide a new visual experience that differs from that of conventional flat screens.

[0076] Example 1

[0077] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0078] Conventional video projection technology has difficulty adapting existing flat images to a 360-degree spherical screen, making it difficult to maintain the continuity and smoothness of the image. Furthermore, there are limited ways for users to select specific scenes and determine the optimal display method. This makes it difficult to provide a highly immersive visual experience.

[0079] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0080] In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, a generating AI means for filling in the gaps between each grid-shaped division, means for the user to select a specific scene, means for converting the video into a 360-degree full screen or a central rectangular display depending on the selected scene, means for integrating and saving the final video for spherical projection, and means for projecting the integrated video onto a spherical screen. This makes it possible to smoothly convert existing video for spherical projection and provide the optimal display method based on the user's selection.

[0081] The "means for acquiring the specified video data" refers to the means used by the server to read and process the video file specified by the user.

[0082] The "means for dividing image data into a grid" is a means for dividing acquired image data based on a certain mesh size, making it easier to process the data individually.

[0083] The "generative AI means for filling in the gaps between each grid-like divided area" is a means for using a generative AI model to fill in the gaps that occur between divided meshes, thereby maintaining the continuity and smoothness of the image.

[0084] The "means for the user to select a particular scene" refers to an interface or operation means for the user to select a particular scene or location that he or she wishes to display.

[0085] The "means for converting the video into a 360-degree full screen or a rectangular display in the center according to the selected scene" is a means for converting the display format of the video into a 360-degree full screen or a rectangular display in the center based on the user's selection.

[0086] The "means for integrating and storing final spherical projection images" refers to a means for integrating interpolated and transformed image frames and storing them in a format suitable for a spherical screen.

[0087] The "means for projecting the integrated image onto a spherical screen" refers to a means for projecting the integrated and stored image onto a spherical screen using a projection device.

[0088] This invention is a system that uses a generative AI model to convert existing video data into images that can be projected onto a 360-degree spherical screen. This system is composed of a server, a terminal, and a user working together, and each step plays a different role.

[0089] Server Roles

[0090] Acquiring video data

[0091] The server retrieves the video data file specified by the user. For example, if the user specifies a file called "movie.mp4" through the interface, the server reads this file from the file system and saves it in a temporary directory.

[0092] Mesh division of video

[0093] The server divides the acquired video data into 10x10 meshes for each frame. This process uses an image processing library (e.g., OpenCV). The divided mesh parts are saved in a format that can be processed individually.

[0094] Interpolation between meshes

[0095] The server uses a generative AI model (e.g., GAN or VQ-VAE) to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates a complementary image based on information from surrounding pixels. This results in a smooth transition between the images, making them suitable for spherical projection.

[0096] Scene selection and processing

[0097] The user uses an interface (dedicated software or web application) to select the specific scene they want to view. For example, if the user wants to view an "ocean" scene in full 360 degrees, the server receives this information and generates a video centered on that scene.

[0098] Merging and saving the final footage

[0099] The server merges all processed frames and generates the final spherical projection video file, which is saved in MP4 format and named "converted_movie.mp4" for example.

[0100] Device Role

[0101] Image projection

[0102] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. A projection device (e.g., a 4K projector) connected to the terminal projects the video onto the 360-degree spherical screen, allowing the audience to experience the video in all directions.

[0103] User Roles

[0104] Select a scene

[0105] Through the interface, the user selects a particular scene in the video, and based on this selection, the server performs the appropriate processing for that scene and generates the video.

[0106] Viewing

[0107] Users can view images projected onto a spherical screen in the planetarium. The immersive feeling provided by 360-degree images allows users to enjoy a new visual experience that is different from that of a conventional flat screen.

[0108] Examples and prompts

[0109] For example, if a user wants to use an interface to view a full-screen 360-degree "ocean" scene, they can use the following prompt:

[0110] "Create a full-screen 360-degree ocean scene."

[0111] Based on this prompt, the server uses AI to generate the "ocean" scene, and other scenes (such as indoor scenes) are displayed as a central rectangular image. The generated spherical projection image file is sent from the server to the terminal, which then projects the image onto the spherical screen of the planetarium.

[0112] Through these processing steps, the system of the present invention can easily convert existing images for spherical projection, providing a highly immersive experience.

[0113] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0114] Program processing flow

[0115] Step 1: Acquire video data

[0116] Specific behavior:

[0117] The server reads the video file (e.g., "movie.mp4") specified by the user through the interface. It checks whether the file exists and whether read permissions are correct. The video data is saved in a temporary directory.

[0118] Input and Output:

[0119] Input: User-specified video file name ("movie.mp4")

[0120] Data processing and data calculation: Checking file existence and reading

[0121] Output: Video data is saved to a temporary directory

[0122] Step 2: Mesh division of video data

[0123] Specific behavior:

[0124] The server divides the acquired video data into 10x10 meshes for each frame. This process uses an image processing library (e.g., OpenCV). The divided mesh parts are saved in a format that can be processed individually.

[0125] Input and Output:

[0126] Input: Video data saved in the temporary directory

[0127] Data processing and data calculation: Divide the video frame into 10x10 meshes

[0128] Output: The divided mesh parts are saved as temporary files.

[0129] Step 3: Interpolation between meshes

[0130] Specific behavior:

[0131] The server uses a generative AI model (e.g., GAN or VQ-VAE) to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates a fill-in image based on the surrounding pixel information.

[0132] Input and Output:

[0133] Input: Divided mesh parts

[0134] Data processing and data calculation: Fill gaps between meshes with generated AI

[0135] Output: The completed mesh part is resaved

[0136] Step 4: Select and process the scene

[0137] Specific behavior:

[0138] The user selects the specific scene they want to display using an interface (dedicated software or web application). The server receives this information and generates a video centered on that scene. For example, if you want to display an "ocean" scene in full 360 degrees, the generation AI will perform the interpolation process.

[0139] Input and Output:

[0140] Input: User's scene selection information (e.g., "ocean" scene)

[0141] Data processing and data calculation: Generate images based on selected scenes and process them with AI

[0142] Output: Video data of the processed scene

[0143] Step 5: Merging and saving the final footage

[0144] Specific behavior:

[0145] The server will then combine all the interpolated and processed frames to generate the final spherical projection video file, which will be saved in MP4 format and named "converted_movie.mp4" for example.

[0146] Input and Output:

[0147] Input: Interpolated and processed frames

[0148] Data processing and data calculation: Frame integration and video file generation

[0149] Output: Final spherical projection video file

[0150] Step 6: Projecting the image

[0151] Specific behavior:

[0152] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium using a projection device (e.g., a 4K projector) connected to the terminal.

[0153] Input and Output:

[0154] Input: Final spherical projection image file

[0155] Data processing and data calculation: Playback of video files

[0156] Output: Image projected onto a spherical screen

[0157] Step 7: Select a Scene

[0158] Specific behavior:

[0159] The user selects a particular scene through the interface, and when the user selects a scene and clicks the "View" button on the interface, this information is sent to the server, which processes the scene.

[0160] Input and Output:

[0161] Input: User scene selection information

[0162] Data Processing and Data Computation: Scene-Based Video Processing

[0163] Output: Footage of the processed scene

[0164] Step 8: Viewing

[0165] Specific behavior:

[0166] Users can enjoy the 360-degree immersive experience by viewing images projected onto a spherical screen in a planetarium.

[0167] Input and Output:

[0168] Input: Image projected onto a spherical screen

[0169] Data processing and calculation: None

[0170] Output: The immersive visual experience the user experiences

[0171] In this way, the system performs appropriate processing at each step to adapt existing footage to a 360-degree spherical screen, providing users with a new visual experience.

[0172] (Application example 1)

[0173] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0174] Conventional video streaming services require special equipment and complex operations to enhance the user's sense of immersion, often limiting the quality of the visual experience. Furthermore, there is a need for an efficient method to convert existing video data into 360-degree spherical video and enable real-time viewing.

[0175] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0176] In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, a generation AI means for filling in gaps between each grid-shaped division, means for the user to select a specific scene, means for converting the video into a 360-degree full screen or rectangular display depending on the selected scene, means for integrating and saving the final video for spherical projection, means for streaming the final video for spherical projection to the user's terminal, and means for performing video data interpolation processing based on a 10x10 mesh. This allows users to convert existing video data into high-quality 360-degree spherical video and enjoy an immersive visual experience in real time without the need for special equipment or complex operations.

[0177] The "means for acquiring the designated video data" is a means by which the server receives the video data file that the user wants to view and starts processing.

[0178] The "means for dividing image data into a grid" is a means for dividing the acquired image data into a grid of a certain size and saving each mesh in a processable form.

[0179] The "generative AI means for filling in the gaps between each grid-like divided area" is a means for using a generative AI to analyze the gaps between divided meshes and generate a filled image based on surrounding pixel information.

[0180] The "means for the user to select a particular scene" is a means for the user to select a particular scene that he or she wishes to display through an interface.

[0181] "Means for converting the video into a 360-degree full screen or rectangular display according to the selected scene" refers to means for converting the video into a 360-degree full screen display or a central rectangular display based on the selected scene.

[0182] The "means for integrating and saving the final spherical projection image" is a means for integrating all the processed frames and saving them on a server as a final spherical projection image file.

[0183] The "means for projecting an image onto a spherical screen" refers to a means for projecting the final image received by the terminal onto a spherical screen, thereby displaying an image that spreads in all directions.

[0184] The "means for streaming the final spherical projection image to the user's terminal" refers to a means for transmitting the final image generated by the server to the user's terminal in real time, enabling continuous playback.

[0185] "Means for performing video data interpolation processing based on a 10x10 mesh" refers to a means for dividing a video frame into a 10x10 mesh and using a generation AI to interpolate the gaps between the meshes.

[0186] This invention is a system that uses generative AI to convert existing video data into video that can be projected onto a 360-degree spherical screen, enabling real-time viewing on a smartphone. The system is composed of a server, terminals, and users working together.

[0187] Server Roles

[0188] Acquiring video data

[0189] The server retrieves the specified video data file. For example, when a user specifies the file name of a movie or video they want to watch, the server reads the video data and begins processing.

[0190] Mesh division of video

[0191] The server divides the acquired video data into a 10x10 grid. Each mesh is saved in a format that can be processed individually. This allows subsequent processing by the generation AI to be performed efficiently.

[0192] Interpolation between meshes

[0193] The server uses a generative AI model to fill in the gaps between the divided meshes. The generative AI analyzes the boundaries of each mesh and generates a complementary image based on surrounding pixel information. This results in a smooth transition between images, making them suitable for spherical projection.

[0194] Scene selection and processing

[0195] The user selects a specific scene (e.g., ocean or space) through the interface. Based on this selection, the server displays the specific scene as a full 360-degree screen and processes the other scenes as a rectangular display in the center.

[0196] Merging and saving the final footage

[0197] The server then combines all the processed frames to generate the final spherical projection video file, which is then stored on the server and subsequently streamed to the user's device in real time.

[0198] Device Role

[0199] Projecting and viewing images

[0200] The device receives the final video file sent from the server and displays it as a 360-degree spherical image, allowing users to view images in all directions using their smartphones. This provides a high-quality immersive experience without the need for special equipment.

[0201] User Roles

[0202] Select a scene

[0203] Users select the specific scenes they want to watch through dedicated software or a web application, and the server performs the necessary video processing based on this selection.

[0204] Watching videos

[0205] Users can watch 360-degree spherical images streamed in real time on their smartphones, providing a new visual experience that is different from that of conventional flat screens.

[0206] Hardware and software used

[0207] Hardware: Servers, smartphones

[0208] Software: Python, OpenCV, MoviePy, FastAI

[0209] Data processing and calculation

[0210] Data Acquisition and Meshing

[0211] Using Python and OpenCV, we take the video data and divide it into a 10x10 mesh.

[0212] Interpolation between meshes

[0213] Using FastAI, the generative AI model complements the mesh boundaries of the video frame to generate smooth images.

[0214] Video integration and storage

[0215] The final video is integrated and saved using OpenCV and MoviePy, and data for streaming distribution is generated.

[0216] Examples and prompts

[0217] As a concrete example, let's say a user wants to display an "ocean" scene in full 360-degree screen. The user selects the "ocean" scene using the interface. Based on this information, the server uses AI to generate the "ocean" scene for full 360-degree screen display, and processes other scenes (e.g., indoor scenes) as rectangular displays in the center.

[0218] Prompt Sentence Examples

[0219] "Divide the following video data into a 10x10 mesh, and use a generative AI model to fill in the gaps between the meshes to generate an image for 360-degree spherical projection."

[0220] This system allows users to easily enjoy high-quality 360-degree spherical images.

[0221] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0222] Step 1:

[0223] Acquiring video data

[0224] The server retrieves the video data file (e.g., movie.mp4) specified by the user. Specifically, the user specifies the video file they want to watch, and the server reads that file. In this process, the path and name of the video file are input data, and the retrieved video is output.

[0225] Step 2:

[0226] Mesh division of video

[0227] The server divides the acquired video data into a grid. Specifically, it uses Python's OpenCV library to divide the video frame into a 10x10 mesh and saves each mesh in a processable format. The input of this step is the acquired video frame, and the output is each divided mesh.

[0228] Step 3:

[0229] Interpolation between meshes

[0230] The server uses a generative AI model to fill in the gaps between the meshes. Specifically, it uses the FastAI library to have the generative AI model analyze the boundaries of each mesh and generate a filled image based on the surrounding pixel information. The input to this step is the segmented mesh data, and the output is the filled image data.

[0231] Step 4:

[0232] Select a scene

[0233] The user selects a specific scene through the interface. Specifically, the user selects the scene they want to display (e.g., "ocean" or "space") using dedicated software or a web application, and this information is sent to the server. The input of this step is the user's selection information, and the output is the selected scene information.

[0234] Step 5:

[0235] Scene-specific video processing

[0236] The server processes the video according to the selected scene. Specifically, it converts certain scenes into a 360-degree full-screen display based on the selection information, and other scenes into a rectangular display with the selection information at the center. The video conversion is performed using Python and OpenCV. The input of this step is the selected scene information and the interpolated video data, and the output is the converted video data.

[0237] Step 6:

[0238] Merging and saving the final footage

[0239] The server merges all processed video frames to generate the final spherical projection video file. Specifically, it uses OpenCV and MoviePy to merge the frames into a single video file. The input of this step is the converted video data, and the output is the merged final video file.

[0240] Step 7:

[0241] Video streaming

[0242] The server then streams the final video file to the user's device. Specifically, it uses HTTP streaming technology to send the video to the user's smartphone in real time. The input of this step is the final video file, and the output is the video played on the user's device.

[0243] Step 8:

[0244] Projecting and viewing images

[0245] The user's device displays the final video received from the server as a 360-degree spherical video. Specifically, a smartphone application plays the video, allowing the user to view the omnidirectional video without the need for special equipment. The input of this step is the streamed video data, and the output is the user's visual experience.

[0246] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0247] This invention is a system that uses generative AI to convert existing video data into video that can be projected onto a 360-degree spherical screen, and combines it with an emotion engine to realize a video display that responds to the user's emotions. This system is composed of a server, terminals, and users working together, and each step plays a different role.

[0248] Server Roles

[0249] Acquiring video data

[0250] The server retrieves and reads the specified video data file. For example, if the file name is movie.mp4, the server reads this file and starts processing.

[0251] Mesh division of video

[0252] The server divides the acquired video data into a grid. Specifically, it divides the video frame into 10x10 meshes and saves each mesh as separate data.

[0253] Interpolation between meshes

[0254] The server uses generative AI to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates pixels to fill in the gaps based on information from surrounding pixels. This filling process smooths the connections between the meshes, generating a continuous image suitable for spherical projection.

[0255] Emotion engine integration

[0256] The server is integrated with an emotion engine that recognizes the user's emotions. This emotion engine analyzes changes in the user's facial expressions and voice to recognize emotions in real time, and automatically adjusts the video scene selection and display format based on that information.

[0257] Scene selection and processing

[0258] In addition to the user selecting a specific scene, the emotion engine monitors the user's emotions in real time, and based on this information, the video will either switch to a full 360-degree screen view or adjust to a central rectangular view depending on the emotion.

[0259] Merging and saving the final footage

[0260] The server then combines all the processed frames to generate the final spherical projection video file. The resulting video file is saved on the server in the format converted_movie.mp4, for example. To do this, a video processing library is used to connect the frames and output them as a single continuous video file.

[0261] Device Role

[0262] Image projection

[0263] The device receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. The video is enlarged and displayed on a 360-degree spherical screen using a projector connected to the device. The video display format is automatically adjusted based on information from the emotion engine.

[0264] User Roles

[0265] Select a scene

[0266] Users can operate the interface to select specific scenes in the video. They can also check their own emotional state through the interface. The emotion engine analyzes the user's facial expressions and voice and provides the optimal video display format.

[0267] Viewing

[0268] Users view images projected onto a spherical screen in the planetarium. The emotion engine recognizes emotions in real time, and if the user is feeling emotional, it switches to a full-screen 360-degree display of a specific scene. In this way, users can enjoy a highly immersive visual experience.

[0269] Specific examples

[0270] For example, if a user is viewing a 360-degree full-screen "ocean" scene in a planetarium, the emotion engine will analyze the user's facial expressions and, if it detects signs of emotion, automatically switch to a full-screen 360-degree display of the next scene. If the emotion engine's analysis determines that the user is bored, it will switch to a rectangular display in the center and provide other content. This process ensures that the user always receives the optimal visual experience.

[0271] Through these processing steps, the system of the present invention can easily convert existing images for spherical projection, providing a highly immersive experience. Furthermore, by integrating an emotion engine, it is possible to realize image display that adapts to the user's emotions.

[0272] The processing flow will be explained below.

[0273] Step 1: Acquire video data

[0274] The server retrieves the specified video data file. Specifically, if the file name is movie.mp4, the server reads this file and uses a video processing library (e.g., OpenCV) to store each frame in a buffer.

[0275] Step 2: Meshing the image

[0276] The server divides each loaded frame into a grid. Specifically, it divides each frame into 10x10 meshes and saves each mesh as separate data. This separates the fine details of the image, making it easier to perform subsequent interpolation processing.

[0277] Step 3: Interpolation between meshes

[0278] The server uses generative AI to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates pixels to fill in the gaps based on information from surrounding pixels. This step smooths the connections between the meshes, resulting in a continuous image suitable for spherical projection.

[0279] Step 4: Obtaining emotion data

[0280] The emotion engine integrated into the server acquires the user's facial expression and voice data to recognize the user's emotions in real time. This involves capturing the user's facial expressions and voice using a camera and microphone, and analyzing the emotions using emotion recognition algorithms.

[0281] Step 5: Select a Scene

[0282] The user operates the interface to select a specific scene. When the user selects a scene, the emotion engine's analysis results are also taken into account. For example, if the user is emotional, the emotion engine will automatically suggest an enhanced scene display.

[0283] Step 6: Decide on the scene display format

[0284] The server determines the display format for each scene based on the scene information selected by the user and the analysis results of the emotion engine. Specifically, if the server recognizes that the user is moved, it displays a specific scene in full 360-degree screen, spreading the image in all directions instead of displaying it in a rectangular format in the center.

[0285] Step 7: Merging and saving the final footage

[0286] The server then combines all the processed frames to generate the final spherical projection video file (e.g., converted_movie.mp4). The resulting video file is then concatenated using a video processing library to create a single continuous video file, which is then saved on the server.

[0287] Step 8: Projecting the image

[0288] The device receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. Using a projector connected to the device, the video is expanded and displayed on a 360-degree spherical screen. The video is optimally adjusted based on the analysis of the emotion engine.

[0289] Step 9: Watch the video

[0290] Users watch images projected onto a spherical screen in the planetarium. The emotion engine analyzes the user's emotions in real time, adjusting the projection of specific scenes if the user is excited, for example, to optimize the user experience. In this way, users can enjoy a highly immersive visual experience.

[0291] Example 2

[0292] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0293] Conventional video display systems have had difficulty optimizing video display in response to user emotions. Furthermore, complex data processing and advanced technology are required to project existing video data in a form suitable for a spherical screen. This can result in a degradation of the quality of the user experience. The purpose of this invention is to solve these problems and realize a highly immersive video display that responds to the user's emotions.

[0294] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, and a generation AI means for filling in gaps between each grid-shaped division. This makes it possible to process existing video data into an image suitable for spherical projection. Furthermore, by including means for a user to select a specific scene, means for converting the image into a 360-degree full screen or rectangular display depending on the selected scene, means for integrating and saving the final image for spherical projection, and means for projecting the image onto the spherical screen, it is possible to realize an image display that is optimal for the scene and emotion specified by the user. Furthermore, by including emotion engine means for recognizing the user's emotion and means for automatically adjusting the image display format according to the user's emotion using the emotion engine means, it is possible to realize real-time adjustment of the image display according to the user's emotion.

[0295] Definitions of important words

[0296] "Specified video data" refers to a video or video file specified by a user or the system prior to entering the system.

[0297] "Generative AI means" refers to a method or device that uses artificial intelligence technology to fill in gaps or missing parts of video data and generate smooth, continuous video.

[0298] "User selection means" refers to a method or device that allows a user to select a particular scene or video portion from the interface.

[0299] "Emotion engine means" refers to technology or a device that analyzes a user's facial expressions, voice, and other biometric information to recognize the user's emotional state in real time.

[0300] "Means for transforming video into a 360-degree full-screen or rectangular display" refers to technology or devices that adapt video data to an all-around display format or a specific shape based on the user's visual preferences.

[0301] "Means for integrating and saving the final spherical projection image" refers to the technology and equipment for integrating and saving the results of each process into a single continuous image file.

[0302] "Means for projecting an image onto a spherical screen" refers to a method or device for displaying the generated spherical projection image on a screen capable of displaying 360 degrees.

[0303] "Means for automatically adjusting the video display format according to the user's emotions using emotion engine means" refers to technology or devices that dynamically change the video display format based on the results of user emotion recognition, providing an optimal visual experience.

[0304] MODE FOR CARRYING OUT THE INVENTION

[0305] This invention is a system that uses a generative AI model to convert existing video data into video that can be projected onto a 360-degree spherical screen, and combines it with an emotion engine to realize video display that responds to the user's emotions. This system is composed of a server, terminals, and users working together, and the roles of each are explained in detail below.

[0306] Server Roles

[0307] Acquiring video data

[0308] The server retrieves and reads the specified video data file. Specifically, if the user specifies a file called "movie.mp4", the server reads the file from a specified directory and loads it into memory as byte data. This retrieval process uses standard file I / O operations and byte stream processing libraries (e.g., Python's os module and io module).

[0309] Mesh division of video

[0310] The server divides the acquired video data into a grid for each frame. Specifically, it uses an image processing library (e.g., OpenCV library) to divide each frame into a 10x10 mesh. Each mesh portion is temporarily saved as individual small image data. This allows the next interpolation process to proceed smoothly.

[0311] Interpolation between meshes

[0312] The server uses a generative AI model to fill in the gaps between each divided mesh. To do this, the server uses a generative AI model (e.g., GANs or AutoEncoder) to analyze surrounding pixel information and generate pixels for the gaps. Specifically, the server trains the model using a deep learning framework such as PyTorch or TENSORFLOW (registered trademark), and outputs the filled-in parts based on the boundary pixel information of each mesh. This filling process smooths the connections between the meshes.

[0313] Emotion engine integration

[0314] The server is integrated with an emotion engine that analyzes facial and voice data sent from the user's webcam and microphone. This analysis uses face recognition APIs and voice analysis libraries (e.g., OpenCV and Dlib, Google® Cloud Speech-to-Text API). The analyzed emotion information is used to adjust the video display format.

[0315] Scene selection and processing

[0316] The server selects an appropriate video scene based on the user's interface operations and data from the emotion engine. For example, if the user selects an "ocean scene" on the interface, the server obtains frame data for that scene, and if the emotion engine detects the user's emotion, it changes the video to 360-degree full-screen mode.

[0317] Merging and saving the final footage

[0318] The server reintegrates all meshes after the interpolation process is completed to generate the final spherical projection video file. It then uses a video processing library (e.g., FFmpeg) to concatenate each frame and save it on the server as a single continuous video file (e.g., converted_movie.mp4). This process ensures that all frames are displayed continuously.

[0319] Device Role

[0320] Image projection

[0321] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. The image is enlarged using a 360-degree projector (e.g., a fisheye projector) connected to the terminal. The display format of the image is automatically adjusted according to the user's emotions, providing the user with an enhanced visual experience.

[0322] User Roles

[0323] Select a scene

[0324] The user operates the interface to select a specific scene. The user can also check their own emotional state through the interface. The emotion engine analyzes the user's facial expressions and voice and provides a video display format that best suits the user's emotions.

[0325] Viewing

[0326] Users view images projected onto a spherical screen in a planetarium. The emotion engine monitors the user's emotions in real time, and if the user is emotionally moved, the scene switches to a full 360-degree screen. This process allows users to view the images with a high level of immersion.

[0327] Examples and prompts

[0328] For example, consider a case where a user is viewing an "ocean scene" in a planetarium while viewing it in full 360-degree screen. If the emotion engine analyzes the user's facial expression and recognizes signs of emotion, it automatically switches to displaying the next scene in full 360-degree screen. Conversely, if it determines that the user is bored, it switches the image to a rectangular display in the center and provides other content.

[0329] An example prompt is, "Convert the movie file movie.mp4 into a 360-degree video and use the emotion engine to adjust the display format according to the user's emotion."

[0330] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0331] Program processing flow

[0332] Step 1: Acquire video data

[0333] The server retrieves the specified video data file (input: specified file path). Based on the prompt, it retrieves the specified file (e.g., movie.mp4) from the directory and loads it into memory as byte data. Specifically, it opens the file using Python's os and io modules and reads its contents as a data stream. The output is the video data loaded into memory.

[0334] Step 2: Meshing the image

[0335] The server divides the acquired video data into a grid for each frame (input: video data in memory). Specifically, it uses an image processing library (e.g., OpenCV) to divide each frame into a 10x10 mesh. Each mesh portion is temporarily saved as individual image data. Specifically, it divides the frame into an array, and then divides it into a grid using a cv2 function. The output is a list of each divided mesh image.

[0336] Step 3: Interpolation between meshes

[0337] The server uses a generative AI model to fill in the gaps between each divided mesh (input: list of divided mesh images). Using a generative AI model (e.g., GANs or AutoEncoder), it analyzes surrounding pixel information and generates pixels in the gaps. Specifically, it uses PyTorch or TensorFlow to input the boundary data of each mesh into the AI ​​model, which then outputs the filled pixel information. The output is a continuous mesh image with the gaps filled in.

[0338] Step 4: Integrating the Emotion Engine

[0339] The server integrates an emotion engine that analyzes the user's emotions in real time (input: user's facial expression data and voice data). The emotion engine uses face recognition APIs and voice analysis libraries (e.g., OpenCV and Dlib, Google Cloud Speech-to-Text API) to analyze the user's emotional state. The emotion data is used to adjust the video display. Specifically, it analyzes real-time streams from the webcam and microphone and outputs the type and intensity of emotions as data. The output is the analyzed emotion information.

[0340] Step 5: Select and process the scene

[0341] The server selects and processes video scenes based on user operations and data from the emotion engine (input: user-selected data and emotion information). Specifically, it obtains the scene selected in the user interface (e.g., "ocean scene") and switches to a 360-degree full-screen or rectangular display depending on the information from the emotion engine. Specifically, it analyzes the emotion data and dynamically changes the video display format. The output is video data adjusted to the optimal display format.

[0342] Step 6: Merging and saving the final footage

[0343] The server reintegrates all meshes after completion of the interpolation process and generates the final spherical projection video file (input: list of interpolated mesh images). It uses a video processing library (e.g., FFmpeg) to concatenate each frame and save it as a single continuous video file on the server. Specifically, it concatenates and encodes the frames and saves the result in an appropriate file format (e.g., converted_movie.mp4). The output is the saved spherical projection video file.

[0344] Step 7: Projecting the image

[0345] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium (input: final video file). A 360-degree compatible projector (e.g., a Fisheye projector) is used to enlarge and display the image. Specifically, the projector's control software is used to decode the video file and project it onto the screen. The output is a 360-degree image displayed on the spherical screen.

[0346] Step 8: Select and watch a scene

[0347] The user operates the interface to select a specific scene and view it in the planetarium (input: user selection). The emotion engine monitors the user's emotions in real time and switches between full-screen and rectangular display depending on the emotion. Specifically, the scene display is adjusted based on the user's operations and emotional data. The output is the visual experience the user views.

[0348] (Application example 2)

[0349] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0350] Conventional systems that project video data onto a spherical screen have the problem of being unable to display images adaptively according to the user's emotions. Furthermore, in industrial environments, there has been a lack of systems that dynamically highlight important information when operators monitor the status of machinery. This has led to problems such as increased operator stress and reduced work efficiency.

[0351] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0352] In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, a generation AI means for filling in gaps between each grid-shaped division, means for the user to select a specific scene, means for converting the video into a 360-degree full screen or rectangular display depending on the selected scene, means for integrating and saving the final video for spherical projection, means for projecting the video onto the spherical screen, means for recognizing the user's emotions using an emotion engine and adaptively displaying information accordingly, and means for displaying the video on a head-mounted display in an industrial environment and dynamically emphasizing important information. This enables adaptive video display according to the user's emotions, improving the efficiency of operator monitoring work in industrial environments and reducing stress.

[0353] The "means for acquiring designated video data" is a method for acquiring and reading a video data file designated in advance by the user or the system.

[0354] The "means for dividing video data into a grid" is a method for dividing acquired video data into a fixed grid pattern and storing the data as individual grids.

[0355] "Generative AI means" refers to artificial intelligence technology that analyzes the gaps between divided grid-like areas and fills them in based on surrounding pixel information.

[0356] The "means for the user to select a particular scene" is a method in which the user operates the interface to select a particular video scene that he or she wishes to view.

[0357] "Means for converting the video to a 360-degree full screen or rectangular display depending on the selected scene" refers to a method for converting the video so that the scene selected by the user is displayed in a 360-degree full screen or a rectangular display in the center.

[0358] The "means for integrating and saving the final spherical projection image" is a method for combining the generated image frames into a single continuous file and saving it in a format that can be projected onto a spherical screen.

[0359] "Means for projecting images onto a spherical screen" refers to a method using a projector or hardware for projecting a stored image file onto a spherical screen.

[0360] An "emotion engine" is a technology that analyzes a user's facial expressions and voice and recognizes their emotional state in real time.

[0361] "Means for adaptively displaying information" refers to a method for dynamically adjusting the information and images displayed based on the user's emotional state.

[0362] "Industrial environment" refers to an environment where industrial activities take place, such as a factory or manufacturing site.

[0363] A "head-mounted display" is a device worn on the user's head that displays images and information directly into the user's field of vision.

[0364] The "means for dynamically highlighting important information" is a method for detecting important information in an industrial environment in real time and highlighting it to the user.

[0365] This invention is a system that uses a head-mounted display (HMD) in an industrial environment to visualize images in a factory in 360 degrees and dynamically highlight information according to the operator's emotions. An embodiment of this system will be described in detail below.

[0366] Server Roles

[0367] The server performs the following main functions:

[0368] Acquiring video data

[0369] The server acquires and reads the specified video data file, using a video processing library such as OpenCV or FFmpeg.

[0370] Mesh division of video

[0371] The server divides the acquired video data into a grid. Specifically, it divides the video frame into 10x10 meshes and saves each mesh as separate data.

[0372] Interpolation between meshes

[0373] The server uses generative AI to fill in the gaps between the divided meshes. Using a generative AI model (such as GPT-3 (registered trademark) 5), it analyzes the boundaries of each mesh and generates pixels for the gaps based on information from surrounding pixels. This filling process smooths the connections between the meshes, generating a continuous image suitable for 360-degree projection.

[0374] Emotion engine integration

[0375] The server is integrated with an emotion engine that recognizes the user's emotions. For example, using Azure® Cognitive Services or TensorFlow, it analyzes changes in the user's facial expressions and voice to recognize emotions in real time. Based on this information, the video scene selection and display format are automatically adjusted.

[0376] Scene selection and processing

[0377] In addition to the user selecting a specific scene, the emotion engine monitors the user's emotions in real time, and based on this information, the video will either switch to a full 360-degree screen view or adjust to a central rectangular view depending on the emotion.

[0378] Merging and saving the final footage

[0379] The server then combines all the processed frames to generate the final 360-degree projection video file. The resulting video file is saved on the server in a format such as converted_movie.mp4. This is done by using a video processing library to connect the frames and output them as a single continuous video file.

[0380] Device Role

[0381] Image projection

[0382] The device receives the final video file sent from the server and displays it on the head-mounted display. The video is displayed in 360 degrees using the display device connected to the device. The video display format is automatically adjusted based on information from the emotion engine.

[0383] User Roles

[0384] Select a scene

[0385] Users can operate the interface to select specific scenes in the video. They can also check their own emotional state through the interface. The emotion engine analyzes the user's facial expressions and voice and provides the optimal video display format.

[0386] Viewing

[0387] The user wears a head-mounted display and watches a 360-degree visualization. The emotion engine recognizes emotions in real time and can highlight important information and issue warnings if the user is feeling stressed.

[0388] Specific examples

[0389] For example, if an operator is monitoring machinery in a factory and the emotion engine detects that the operator's stress level is high, the HMD image will instantly change based on that information. A 360-degree visualization of the machine that detected the abnormality will be displayed, highlighting the important parts. Below is an example of a prompt using a generative AI model:

[0390] "What is the best way to acquire real-time video data from machines in this factory and display it appropriately based on the operator's emotions?"

[0391] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0392] Step 1:

[0393] Acquiring video data

[0394] The server retrieves and reads the specified video data file. A video file name such as movie.mp4 is given as input data. The server reads this file using a video processing library such as OpenCV or FFmpeg and obtains the video frames. The output is the image data for each frame.

[0395] Step 2:

[0396] Grid division of video data

[0397] The server divides the captured video frame into a grid pattern. The input data is the video frame captured in the previous step. The video frame is divided into 10x10 meshes, and each mesh is saved as separate data. The output is the data for each mesh area.

[0398] Step 3:

[0399] Interpolation between meshes

[0400] The server uses a generative AI method to fill in the gaps between each divided mesh region. The input data is the data for the divided mesh regions. Using a generative AI model (e.g., GPT-3.5), it analyzes the boundaries of each mesh and generates pixels in the gaps based on the surrounding pixel information. The output is the mesh region data after the completion of the filling process.

[0401] Step 4:

[0402] Emotion recognition

[0403] The server uses an emotion engine to recognize the user's emotions in real time. The input data is the user's facial expression and voice data. The emotion engine (e.g., Azure Cognitive Services) analyzes the data and recognizes the user's emotional state. The output is the recognized emotional state data.

[0404] Step 5:

[0405] Select a scene

[0406] The user operates the interface to select a specific scene. The input data is a list of scenes. For each scene selected by the user, the server identifies the video data for that scene. The output is the video data for the selected scene.

[0407] Step 6:

[0408] Video display format conversion

[0409] The server converts the video into a 360-degree full-screen or central rectangular display depending on the selected scene. The input data is the video data of the selected scene and the emotional state data recognized in the previous step. The server converts the video into the optimal display format based on the emotional state. The output is the video data in the converted display format.

[0410] Step 7:

[0411] Merging and saving the final footage

[0412] The server combines and saves the converted video frames as a single continuous file. The input data is the converted video frame data. Using a video processing library, it concatenates these frames and saves them in a format such as converted_movie.mp4. The output is the final video file.

[0413] Step 8:

[0414] Image projection

[0415] The terminal receives the final video file sent from the server and displays it on the head-mounted display. The input data is the saved final video file. This data is sent to the HMD, which displays a 360-degree visualized video. The output is the video displayed on the HMD.

[0416] Step 9:

[0417] User reviews

[0418] The user wears a head-mounted display and watches a dynamically adjusted 360-degree video. The input data is the video displayed on the HMD. The output is the user's emotional response, which is then fed back to the emotion engine.

[0419] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0420] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0421] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0422] [Second embodiment]

[0423] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0424] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0425] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0426] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0427] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0428] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0429] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0430] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0431] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0432] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0433] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0434] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0435] This invention is a system that uses generative AI to convert existing video data into images that can be projected onto a 360-degree spherical screen. This system is composed of a server, terminals, and users working together, and each step plays a different role.

[0436] Server Roles

[0437] Acquiring video data

[0438] The server retrieves the specified video data file. For example, if a video file named movie.mp4 is specified, the server reads it and starts processing.

[0439] Mesh division of video

[0440] The server divides the acquired video data into a grid. For example, it divides a video frame into a 10x10 mesh and saves each mesh in a format that can be processed individually.

[0441] Interpolation between meshes

[0442] The server uses generation AI to fill in the gaps between the divided meshes. The generation AI analyzes the boundaries of each mesh and generates a complementary image based on information from surrounding pixels. This results in a smooth transition between images, making it suitable for spherical projection.

[0443] Scene selection and processing

[0444] Through the interface, the user selects whether to display a particular scene (e.g., ocean or space) in full 360-degree screen or in a central rectangular view. This selection information is sent to the server, which then performs the appropriate processing for each scene.

[0445] Merging and saving the final footage

[0446] The server merges all the processed frames and generates the final spherical projection video file, which is saved on the server in the format converted_movie.mp4, for example.

[0447] Device Role

[0448] Image projection

[0449] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. Using a projector connected to the terminal, the video is expanded and displayed on a 360-degree spherical screen, allowing the audience to experience the video in all directions.

[0450] User Roles

[0451] Select a scene

[0452] The user selects a specific scene from the video through an interface (e.g., dedicated software or a web application). Based on this selection, the system determines whether that scene should be displayed as a full 360-degree screen, while the other scenes should be displayed as a rectangular area in the center.

[0453] Viewing

[0454] Users can view images projected onto a spherical screen in the planetarium. The immersive feeling provided by 360-degree images allows users to enjoy a new visual experience that is different from that of a conventional flat screen.

[0455] Specific examples

[0456] For example, if a user wants to view an "ocean" scene in full 360-degree screen, the user selects the "ocean" scene through the interface. Based on this information, the server uses AI to generate the "ocean" scene for full 360-degree screen viewing, while other scenes (e.g., indoor scenes) are displayed as rectangular images in the center. Finally, the generated spherical projection video file is sent from the server to the device and projected onto the spherical screen of the planetarium. This process allows the user to enjoy a new visual experience.

[0457] Through these processing steps, the system of the present invention can easily convert existing images for spherical projection, providing a highly immersive experience.

[0458] The processing flow will be explained below.

[0459] Step 1: Acquire video data

[0460] The server retrieves the specified video data file. For example, if the file name is movie.mp4, the server reads this file and starts processing it. The video file is split into frames using a video processing library, and each frame is saved in a buffer.

[0461] Step 2: Meshing the image

[0462] The server divides each loaded frame into a grid. Specifically, it divides the frame into 10x10 meshes and treats each mesh as individual data. This process is performed using an image processing library.

[0463] Step 3: Interpolation between meshes

[0464] The server uses generative AI to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates pixels to fill in the gaps based on information from surrounding pixels. This filling process smooths the connections between the meshes, generating a continuous image suitable for spherical projection.

[0465] Step 4: Select a Scene

[0466] The user operates an interface to select specific scenes in the video, for example, using dedicated software or a web application. For each scene, the user can choose between a full-screen 360-degree view or a central rectangular view.

[0467] Step 5: Decide on the scene display format

[0468] The server receives the scene information selected by the user and determines the display format for each scene based on that information: certain scenes are processed for a full 360-degree screen display, while others are processed for a central rectangular display.

[0469] Step 6: Merging and saving the final footage

[0470] The server then combines all the processed frames to generate the final spherical projection video file. The resulting video file is saved on the server in the format converted_movie.mp4, for example. To do this, a video processing library is used to connect the frames and output them as a single continuous video file.

[0471] Step 7: Projecting the image

[0472] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. Using a projector connected to the terminal, the video is enlarged and displayed on a 360-degree spherical screen. This projection allows the audience to enjoy the video in all directions.

[0473] Step 8: Watch the video

[0474] Users can view images projected onto a spherical screen in the planetarium. The 360-degree images provide a new visual experience that differs from that of conventional flat screens.

[0475] Example 1

[0476] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0477] Conventional video projection technology has difficulty adapting existing flat images to a 360-degree spherical screen, making it difficult to maintain the continuity and smoothness of the image. Furthermore, there are limited ways for users to select specific scenes and determine the optimal display method. This makes it difficult to provide a highly immersive visual experience.

[0478] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0479] In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, a generating AI means for filling in the gaps between each grid-shaped division, means for the user to select a specific scene, means for converting the video into a 360-degree full screen or a central rectangular display depending on the selected scene, means for integrating and saving the final video for spherical projection, and means for projecting the integrated video onto a spherical screen. This makes it possible to smoothly convert existing video for spherical projection and provide the optimal display method based on the user's selection.

[0480] The "means for acquiring the specified video data" refers to the means used by the server to read and process the video file specified by the user.

[0481] The "means for dividing image data into a grid" is a means for dividing acquired image data based on a certain mesh size, making it easier to process the data individually.

[0482] The "generative AI means for filling in the gaps between each grid-like divided area" is a means for using a generative AI model to fill in the gaps that occur between divided meshes, thereby maintaining the continuity and smoothness of the image.

[0483] The "means for the user to select a particular scene" refers to an interface or operation means for the user to select a particular scene or location that he or she wishes to display.

[0484] The "means for converting the video into a 360-degree full screen or a rectangular display in the center according to the selected scene" is a means for converting the display format of the video into a 360-degree full screen or a rectangular display in the center based on the user's selection.

[0485] The "means for integrating and storing final spherical projection images" refers to a means for integrating interpolated and transformed image frames and storing them in a format suitable for a spherical screen.

[0486] The "means for projecting the integrated image onto a spherical screen" refers to a means for projecting the integrated and stored image onto a spherical screen using a projection device.

[0487] This invention is a system that uses a generative AI model to convert existing video data into images that can be projected onto a 360-degree spherical screen. This system is composed of a server, a terminal, and a user working together, and each step plays a different role.

[0488] Server Roles

[0489] Acquiring video data

[0490] The server retrieves the video data file specified by the user. For example, if the user specifies a file called "movie.mp4" through the interface, the server reads this file from the file system and saves it in a temporary directory.

[0491] Mesh division of video

[0492] The server divides the acquired video data into 10x10 meshes for each frame. This process uses an image processing library (e.g., OpenCV). The divided mesh parts are saved in a format that can be processed individually.

[0493] Interpolation between meshes

[0494] The server uses a generative AI model (e.g., GAN or VQ-VAE) to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates a complementary image based on information from surrounding pixels. This results in a smooth transition between the images, making them suitable for spherical projection.

[0495] Scene selection and processing

[0496] The user uses an interface (dedicated software or web application) to select the specific scene they want to view. For example, if the user wants to view an "ocean" scene in full 360 degrees, the server receives this information and generates a video centered on that scene.

[0497] Merging and saving the final footage

[0498] The server merges all processed frames and generates the final spherical projection video file, which is saved in MP4 format and named "converted_movie.mp4" for example.

[0499] Device Role

[0500] Image projection

[0501] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. A projection device (e.g., a 4K projector) connected to the terminal projects the video onto the 360-degree spherical screen, allowing the audience to experience the video in all directions.

[0502] User Roles

[0503] Select a scene

[0504] Through the interface, the user selects a particular scene in the video, and based on this selection, the server performs the appropriate processing for that scene and generates the video.

[0505] Viewing

[0506] Users can view images projected onto a spherical screen in the planetarium. The immersive feeling provided by 360-degree images allows users to enjoy a new visual experience that is different from that of a conventional flat screen.

[0507] Examples and prompts

[0508] For example, if a user wants to use an interface to view a full-screen 360-degree "ocean" scene, they can use the following prompt:

[0509] "Create a full-screen 360-degree ocean scene."

[0510] Based on this prompt, the server uses AI to generate the "ocean" scene, and other scenes (such as indoor scenes) are displayed as a central rectangular image. The generated spherical projection image file is sent from the server to the terminal, which then projects the image onto the spherical screen of the planetarium.

[0511] Through these processing steps, the system of the present invention can easily convert existing images for spherical projection, providing a highly immersive experience.

[0512] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0513] Program processing flow

[0514] Step 1: Acquire video data

[0515] Specific behavior:

[0516] The server reads the video file (e.g., "movie.mp4") specified by the user through the interface. It checks whether the file exists and whether read permissions are correct. The video data is saved in a temporary directory.

[0517] Input and Output:

[0518] Input: User-specified video file name ("movie.mp4")

[0519] Data processing and data calculation: Checking file existence and reading

[0520] Output: Video data is saved to a temporary directory

[0521] Step 2: Mesh division of video data

[0522] Specific behavior:

[0523] The server divides the acquired video data into 10x10 meshes for each frame. This process uses an image processing library (e.g., OpenCV). The divided mesh parts are saved in a format that can be processed individually.

[0524] Input and Output:

[0525] Input: Video data saved in the temporary directory

[0526] Data processing and data calculation: Divide the video frame into 10x10 meshes

[0527] Output: The divided mesh parts are saved as temporary files.

[0528] Step 3: Interpolation between meshes

[0529] Specific behavior:

[0530] The server uses a generative AI model (e.g., GAN or VQ-VAE) to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates a fill-in image based on the surrounding pixel information.

[0531] Input and Output:

[0532] Input: Divided mesh parts

[0533] Data processing and data calculation: Fill gaps between meshes with generated AI

[0534] Output: The completed mesh part is resaved

[0535] Step 4: Select and process the scene

[0536] Specific behavior:

[0537] The user selects the specific scene they want to display using an interface (dedicated software or web application). The server receives this information and generates a video centered on that scene. For example, if you want to display an "ocean" scene in full 360 degrees, the generation AI will perform the interpolation process.

[0538] Input and Output:

[0539] Input: User's scene selection information (e.g., "ocean" scene)

[0540] Data processing and data calculation: Generate images based on selected scenes and process them with AI

[0541] Output: Video data of the processed scene

[0542] Step 5: Merging and saving the final footage

[0543] Specific behavior:

[0544] The server will then combine all the interpolated and processed frames to generate the final spherical projection video file, which will be saved in MP4 format and named "converted_movie.mp4" for example.

[0545] Input and Output:

[0546] Input: Interpolated and processed frames

[0547] Data processing and data calculation: Frame integration and video file generation

[0548] Output: Final spherical projection video file

[0549] Step 6: Projecting the image

[0550] Specific behavior:

[0551] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium using a projection device (e.g., a 4K projector) connected to the terminal.

[0552] Input and Output:

[0553] Input: Final spherical projection image file

[0554] Data processing and data calculation: Playback of video files

[0555] Output: Image projected onto a spherical screen

[0556] Step 7: Select a Scene

[0557] Specific behavior:

[0558] The user selects a particular scene through the interface, and when the user selects a scene and clicks the "View" button on the interface, this information is sent to the server, which processes the scene.

[0559] Input and Output:

[0560] Input: User scene selection information

[0561] Data Processing and Data Computation: Scene-Based Video Processing

[0562] Output: Footage of the processed scene

[0563] Step 8: Viewing

[0564] Specific behavior:

[0565] Users can enjoy the 360-degree immersive experience by viewing images projected onto a spherical screen in a planetarium.

[0566] Input and Output:

[0567] Input: Image projected onto a spherical screen

[0568] Data processing and calculation: None

[0569] Output: The immersive visual experience the user experiences

[0570] In this way, the system performs appropriate processing at each step to adapt existing footage to a 360-degree spherical screen, providing users with a new visual experience.

[0571] (Application example 1)

[0572] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0573] Conventional video streaming services require special equipment and complex operations to enhance the user's sense of immersion, often limiting the quality of the visual experience. Furthermore, there is a need for an efficient method to convert existing video data into 360-degree spherical video and enable real-time viewing.

[0574] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0575] In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, a generation AI means for filling in gaps between each grid-shaped division, means for the user to select a specific scene, means for converting the video into a 360-degree full screen or rectangular display depending on the selected scene, means for integrating and saving the final video for spherical projection, means for streaming the final video for spherical projection to the user's terminal, and means for performing video data interpolation processing based on a 10x10 mesh. This allows users to convert existing video data into high-quality 360-degree spherical video and enjoy an immersive visual experience in real time without the need for special equipment or complex operations.

[0576] The "means for acquiring the designated video data" is a means by which the server receives the video data file that the user wants to view and starts processing.

[0577] The "means for dividing image data into a grid" is a means for dividing the acquired image data into a grid of a certain size and saving each mesh in a processable form.

[0578] The "generative AI means for filling in the gaps between each grid-like divided area" is a means for using a generative AI to analyze the gaps between divided meshes and generate a filled image based on surrounding pixel information.

[0579] The "means for the user to select a particular scene" is a means for the user to select a particular scene that he or she wishes to display through an interface.

[0580] "Means for converting the video into a 360-degree full screen or rectangular display according to the selected scene" refers to means for converting the video into a 360-degree full screen display or a central rectangular display based on the selected scene.

[0581] The "means for integrating and saving the final spherical projection image" is a means for integrating all the processed frames and saving them on a server as a final spherical projection image file.

[0582] The "means for projecting an image onto a spherical screen" refers to a means for projecting the final image received by the terminal onto a spherical screen, thereby displaying an image that spreads in all directions.

[0583] The "means for streaming the final spherical projection image to the user's terminal" refers to a means for transmitting the final image generated by the server to the user's terminal in real time, enabling continuous playback.

[0584] "Means for performing video data interpolation processing based on a 10x10 mesh" refers to a means for dividing a video frame into a 10x10 mesh and using a generation AI to interpolate the gaps between the meshes.

[0585] This invention is a system that uses generative AI to convert existing video data into video that can be projected onto a 360-degree spherical screen, enabling real-time viewing on a smartphone. The system is composed of a server, terminals, and users working together.

[0586] Server Roles

[0587] Acquiring video data

[0588] The server retrieves the specified video data file. For example, when a user specifies the file name of a movie or video they want to watch, the server reads the video data and begins processing.

[0589] Mesh division of video

[0590] The server divides the acquired video data into a 10x10 grid. Each mesh is saved in a format that can be processed individually. This allows subsequent processing by the generation AI to be performed efficiently.

[0591] Interpolation between meshes

[0592] The server uses a generative AI model to fill in the gaps between the divided meshes. The generative AI analyzes the boundaries of each mesh and generates a complementary image based on surrounding pixel information. This results in a smooth transition between images, making them suitable for spherical projection.

[0593] Scene selection and processing

[0594] The user selects a specific scene (e.g., ocean or space) through the interface. Based on this selection, the server displays the specific scene as a full 360-degree screen and processes the other scenes as a rectangular display in the center.

[0595] Merging and saving the final footage

[0596] The server then combines all the processed frames to generate the final spherical projection video file, which is then stored on the server and subsequently streamed to the user's device in real time.

[0597] Device Role

[0598] Projecting and viewing images

[0599] The device receives the final video file sent from the server and displays it as a 360-degree spherical image, allowing users to view images in all directions using their smartphones. This provides a high-quality immersive experience without the need for special equipment.

[0600] User Roles

[0601] Select a scene

[0602] Users select the specific scenes they want to watch through dedicated software or a web application, and the server performs the necessary video processing based on this selection.

[0603] Watching videos

[0604] Users can watch 360-degree spherical images streamed in real time on their smartphones, providing a new visual experience that is different from that of conventional flat screens.

[0605] Hardware and software used

[0606] Hardware: Servers, smartphones

[0607] Software: Python, OpenCV, MoviePy, FastAI

[0608] Data processing and calculation

[0609] Data Acquisition and Meshing

[0610] Using Python and OpenCV, we take the video data and divide it into a 10x10 mesh.

[0611] Interpolation between meshes

[0612] Using FastAI, the generative AI model complements the mesh boundaries of the video frame to generate smooth images.

[0613] Video integration and storage

[0614] The final video is integrated and saved using OpenCV and MoviePy, and data for streaming distribution is generated.

[0615] Examples and prompts

[0616] As a concrete example, let's say a user wants to display an "ocean" scene in full 360-degree screen. The user selects the "ocean" scene using the interface. Based on this information, the server uses AI to generate the "ocean" scene for full 360-degree screen display, and processes other scenes (e.g., indoor scenes) as rectangular displays in the center.

[0617] Prompt Sentence Examples

[0618] "Divide the following video data into a 10x10 mesh, and use a generative AI model to fill in the gaps between the meshes to generate an image for 360-degree spherical projection."

[0619] This system allows users to easily enjoy high-quality 360-degree spherical images.

[0620] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0621] Step 1:

[0622] Acquiring video data

[0623] The server retrieves the video data file (e.g., movie.mp4) specified by the user. Specifically, the user specifies the video file they want to watch, and the server reads that file. In this process, the path and name of the video file are input data, and the retrieved video is output.

[0624] Step 2:

[0625] Mesh division of video

[0626] The server divides the acquired video data into a grid. Specifically, it uses Python's OpenCV library to divide the video frame into a 10x10 mesh and saves each mesh in a processable format. The input of this step is the acquired video frame, and the output is each divided mesh.

[0627] Step 3:

[0628] Interpolation between meshes

[0629] The server uses a generative AI model to fill in the gaps between the meshes. Specifically, it uses the FastAI library to have the generative AI model analyze the boundaries of each mesh and generate a filled image based on the surrounding pixel information. The input to this step is the segmented mesh data, and the output is the filled image data.

[0630] Step 4:

[0631] Select a scene

[0632] The user selects a specific scene through the interface. Specifically, the user selects the scene they want to display (e.g., "ocean" or "space") using dedicated software or a web application, and this information is sent to the server. The input of this step is the user's selection information, and the output is the selected scene information.

[0633] Step 5:

[0634] Scene-specific video processing

[0635] The server processes the video according to the selected scene. Specifically, it converts certain scenes into a 360-degree full-screen display based on the selection information, and other scenes into a rectangular display with the selection information at the center. The video conversion is performed using Python and OpenCV. The input of this step is the selected scene information and the interpolated video data, and the output is the converted video data.

[0636] Step 6:

[0637] Merging and saving the final footage

[0638] The server merges all processed video frames to generate the final spherical projection video file. Specifically, it uses OpenCV and MoviePy to merge the frames into a single video file. The input of this step is the converted video data, and the output is the merged final video file.

[0639] Step 7:

[0640] Video streaming

[0641] The server then streams the final video file to the user's device. Specifically, it uses HTTP streaming technology to send the video to the user's smartphone in real time. The input of this step is the final video file, and the output is the video played on the user's device.

[0642] Step 8:

[0643] Projecting and viewing images

[0644] The user's device displays the final video received from the server as a 360-degree spherical video. Specifically, a smartphone application plays the video, allowing the user to view the omnidirectional video without the need for special equipment. The input of this step is the streamed video data, and the output is the user's visual experience.

[0645] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0646] This invention is a system that uses generative AI to convert existing video data into video that can be projected onto a 360-degree spherical screen, and combines it with an emotion engine to realize a video display that responds to the user's emotions. This system is composed of a server, terminals, and users working together, and each step plays a different role.

[0647] Server Roles

[0648] Acquiring video data

[0649] The server retrieves and reads the specified video data file. For example, if the file name is movie.mp4, the server reads this file and starts processing.

[0650] Mesh division of video

[0651] The server divides the acquired video data into a grid. Specifically, it divides the video frame into 10x10 meshes and saves each mesh as separate data.

[0652] Interpolation between meshes

[0653] The server uses generative AI to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates pixels to fill in the gaps based on information from surrounding pixels. This filling process smooths the connections between the meshes, generating a continuous image suitable for spherical projection.

[0654] Emotion engine integration

[0655] The server is integrated with an emotion engine that recognizes the user's emotions. This emotion engine analyzes changes in the user's facial expressions and voice to recognize emotions in real time, and automatically adjusts the video scene selection and display format based on that information.

[0656] Scene selection and processing

[0657] In addition to the user selecting a specific scene, the emotion engine monitors the user's emotions in real time, and based on this information, the video will either switch to a full 360-degree screen view or adjust to a central rectangular view depending on the emotion.

[0658] Merging and saving the final footage

[0659] The server then combines all the processed frames to generate the final spherical projection video file. The resulting video file is saved on the server in the format converted_movie.mp4, for example. To do this, a video processing library is used to connect the frames and output them as a single continuous video file.

[0660] Device Role

[0661] Image projection

[0662] The device receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. The video is enlarged and displayed on a 360-degree spherical screen using a projector connected to the device. The video display format is automatically adjusted based on information from the emotion engine.

[0663] User Roles

[0664] Select a scene

[0665] Users can operate the interface to select specific scenes in the video. They can also check their own emotional state through the interface. The emotion engine analyzes the user's facial expressions and voice and provides the optimal video display format.

[0666] Viewing

[0667] Users view images projected onto a spherical screen in the planetarium. The emotion engine recognizes emotions in real time, and if the user is feeling emotional, it switches to a full-screen 360-degree display of a specific scene. In this way, users can enjoy a highly immersive visual experience.

[0668] Specific examples

[0669] For example, if a user is viewing a 360-degree full-screen "ocean" scene in a planetarium, the emotion engine will analyze the user's facial expressions and, if it detects signs of emotion, automatically switch to a full-screen 360-degree display of the next scene. If the emotion engine's analysis determines that the user is bored, it will switch to a rectangular display in the center and provide other content. This process ensures that the user always receives the optimal visual experience.

[0670] Through these processing steps, the system of the present invention can easily convert existing images for spherical projection, providing a highly immersive experience. Furthermore, by integrating an emotion engine, it is possible to realize image display that adapts to the user's emotions.

[0671] The processing flow will be explained below.

[0672] Step 1: Acquire video data

[0673] The server retrieves the specified video data file. Specifically, if the file name is movie.mp4, the server reads this file and uses a video processing library (e.g., OpenCV) to store each frame in a buffer.

[0674] Step 2: Meshing the image

[0675] The server divides each loaded frame into a grid. Specifically, it divides each frame into 10x10 meshes and saves each mesh as separate data. This separates the fine details of the image, making it easier to perform subsequent interpolation processing.

[0676] Step 3: Interpolation between meshes

[0677] The server uses generative AI to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates pixels to fill in the gaps based on information from surrounding pixels. This step smooths the connections between the meshes, resulting in a continuous image suitable for spherical projection.

[0678] Step 4: Obtaining emotion data

[0679] The emotion engine integrated into the server acquires the user's facial expression and voice data to recognize the user's emotions in real time. This involves capturing the user's facial expressions and voice using a camera and microphone, and analyzing the emotions using emotion recognition algorithms.

[0680] Step 5: Select a Scene

[0681] The user operates the interface to select a specific scene. When the user selects a scene, the emotion engine's analysis results are also taken into account. For example, if the user is emotional, the emotion engine will automatically suggest an enhanced scene display.

[0682] Step 6: Decide on the scene display format

[0683] The server determines the display format for each scene based on the scene information selected by the user and the analysis results of the emotion engine. Specifically, if the server recognizes that the user is moved, it displays a specific scene in full 360-degree screen, spreading the image in all directions instead of displaying it in a rectangular format in the center.

[0684] Step 7: Merging and saving the final footage

[0685] The server then combines all the processed frames to generate the final spherical projection video file (e.g., converted_movie.mp4). The resulting video file is then concatenated using a video processing library to create a single continuous video file, which is then saved on the server.

[0686] Step 8: Projecting the image

[0687] The device receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. Using a projector connected to the device, the video is expanded and displayed on a 360-degree spherical screen. The video is optimally adjusted based on the analysis of the emotion engine.

[0688] Step 9: Watch the video

[0689] Users watch images projected onto a spherical screen in the planetarium. The emotion engine analyzes the user's emotions in real time, adjusting the projection of specific scenes if the user is excited, for example, to optimize the user experience. In this way, users can enjoy a highly immersive visual experience.

[0690] Example 2

[0691] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0692] Conventional video display systems have had difficulty optimizing video display in response to user emotions. Furthermore, complex data processing and advanced technology are required to project existing video data in a form suitable for a spherical screen. This can result in a degradation of the quality of the user experience. The purpose of this invention is to solve these problems and realize a highly immersive video display that responds to the user's emotions.

[0693] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, and a generation AI means for filling in gaps between each grid-shaped division. This makes it possible to process existing video data into an image suitable for spherical projection. Furthermore, by including means for a user to select a specific scene, means for converting the image into a 360-degree full screen or rectangular display depending on the selected scene, means for integrating and saving the final image for spherical projection, and means for projecting the image onto the spherical screen, it is possible to realize an image display that is optimal for the scene and emotion specified by the user. Furthermore, by including emotion engine means for recognizing the user's emotion and means for automatically adjusting the image display format according to the user's emotion using the emotion engine means, it is possible to realize real-time adjustment of the image display according to the user's emotion.

[0694] Definitions of important words

[0695] "Specified video data" refers to a video or video file specified by a user or the system prior to entering the system.

[0696] "Generative AI means" refers to a method or device that uses artificial intelligence technology to fill in gaps or missing parts of video data and generate smooth, continuous video.

[0697] "User selection means" refers to a method or device that allows a user to select a particular scene or video portion from the interface.

[0698] "Emotion engine means" refers to technology or a device that analyzes a user's facial expressions, voice, and other biometric information to recognize the user's emotional state in real time.

[0699] "Means for transforming video into a 360-degree full-screen or rectangular display" refers to technology or devices that adapt video data to an all-around display format or a specific shape based on the user's visual preferences.

[0700] "Means for integrating and saving the final spherical projection image" refers to the technology and equipment for integrating and saving the results of each process into a single continuous image file.

[0701] "Means for projecting an image onto a spherical screen" refers to a method or device for displaying the generated spherical projection image on a screen capable of displaying 360 degrees.

[0702] "Means for automatically adjusting the video display format according to the user's emotions using emotion engine means" refers to technology or devices that dynamically change the video display format based on the results of user emotion recognition, providing an optimal visual experience.

[0703] MODE FOR CARRYING OUT THE INVENTION

[0704] This invention is a system that uses a generative AI model to convert existing video data into video that can be projected onto a 360-degree spherical screen, and combines it with an emotion engine to realize video display that responds to the user's emotions. This system is composed of a server, terminals, and users working together, and the roles of each are explained in detail below.

[0705] Server Roles

[0706] Acquiring video data

[0707] The server retrieves and reads the specified video data file. Specifically, if the user specifies a file called "movie.mp4", the server reads the file from a specified directory and loads it into memory as byte data. This retrieval process uses standard file I / O operations and byte stream processing libraries (e.g., Python's os module and io module).

[0708] Mesh division of video

[0709] The server divides the acquired video data into a grid for each frame. Specifically, it uses an image processing library (e.g., OpenCV library) to divide each frame into a 10x10 mesh. Each mesh portion is temporarily saved as individual small image data. This allows the next interpolation process to proceed smoothly.

[0710] Interpolation between meshes

[0711] The server uses a generative AI model to fill in the gaps between each divided mesh. To do this, the server uses a generative AI model (e.g., GANs or AutoEncoder) to analyze surrounding pixel information and generate pixels to fill in the gaps. Specifically, the server trains the model using a deep learning framework such as PyTorch or TensorFlow, and outputs the filled-in parts based on the boundary pixel information of each mesh. This filling process smooths the connections between the meshes.

[0712] Emotion engine integration

[0713] The server is integrated with an emotion engine that analyzes facial and voice data sent from the user's webcam and microphone. This analysis uses face recognition APIs and voice analysis libraries (e.g., OpenCV, Dlib, and Google's Cloud Speech-to-Text API). The analyzed emotion information is used to adjust the video display format.

[0714] Scene selection and processing

[0715] The server selects an appropriate video scene based on the user's interface operations and data from the emotion engine. For example, if the user selects an "ocean scene" on the interface, the server obtains frame data for that scene, and if the emotion engine detects the user's emotion, it changes the video to 360-degree full-screen mode.

[0716] Merging and saving the final footage

[0717] The server reintegrates all meshes after the interpolation process is completed to generate the final spherical projection video file. It then uses a video processing library (e.g., FFmpeg) to concatenate each frame and save it on the server as a single continuous video file (e.g., converted_movie.mp4). This process ensures that all frames are displayed continuously.

[0718] Device Role

[0719] Image projection

[0720] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. The image is enlarged using a 360-degree projector (e.g., a fisheye projector) connected to the terminal. The display format of the image is automatically adjusted according to the user's emotions, providing the user with an enhanced visual experience.

[0721] User Roles

[0722] Select a scene

[0723] The user operates the interface to select a specific scene. The user can also check their own emotional state through the interface. The emotion engine analyzes the user's facial expressions and voice and provides a video display format that best suits the user's emotions.

[0724] Viewing

[0725] Users view images projected onto a spherical screen in a planetarium. The emotion engine monitors the user's emotions in real time, and if the user is emotionally moved, the scene switches to a full 360-degree screen. This process allows users to view the images with a high level of immersion.

[0726] Examples and prompts

[0727] For example, consider a case where a user is viewing an "ocean scene" in a planetarium while viewing it in full 360-degree screen. If the emotion engine analyzes the user's facial expression and recognizes signs of emotion, it automatically switches to displaying the next scene in full 360-degree screen. Conversely, if it determines that the user is bored, it switches the image to a rectangular display in the center and provides other content.

[0728] An example prompt is, "Convert the movie file movie.mp4 into a 360-degree video and use the emotion engine to adjust the display format according to the user's emotion."

[0729] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0730] Program processing flow

[0731] Step 1: Acquire video data

[0732] The server retrieves the specified video data file (input: specified file path). Based on the prompt, it retrieves the specified file (e.g., movie.mp4) from the directory and loads it into memory as byte data. Specifically, it opens the file using Python's os and io modules and reads its contents as a data stream. The output is the video data loaded into memory.

[0733] Step 2: Meshing the image

[0734] The server divides the acquired video data into a grid for each frame (input: video data in memory). Specifically, it uses an image processing library (e.g., OpenCV) to divide each frame into a 10x10 mesh. Each mesh portion is temporarily saved as individual image data. Specifically, it divides the frame into an array, and then divides it into a grid using a cv2 function. The output is a list of each divided mesh image.

[0735] Step 3: Interpolation between meshes

[0736] The server uses a generative AI model to fill in the gaps between each divided mesh (input: list of divided mesh images). Using a generative AI model (e.g., GANs or AutoEncoder), it analyzes surrounding pixel information and generates pixels in the gaps. Specifically, it uses PyTorch or TensorFlow to input the boundary data of each mesh into the AI ​​model, which then outputs the filled pixel information. The output is a continuous mesh image with the gaps filled in.

[0737] Step 4: Integrating the Emotion Engine

[0738] The server integrates an emotion engine that analyzes the user's emotions in real time (input: user's facial expression data and voice data). The emotion engine uses face recognition APIs and voice analysis libraries (e.g., OpenCV and Dlib, Google Cloud Speech-to-Text API) to analyze the user's emotional state. The emotion data is used to adjust the video display. Specifically, it analyzes real-time streams from the webcam and microphone and outputs the type and intensity of emotions as data. The output is the analyzed emotion information.

[0739] Step 5: Select and process the scene

[0740] The server selects and processes video scenes based on user operations and data from the emotion engine (input: user-selected data and emotion information). Specifically, it obtains the scene selected in the user interface (e.g., "ocean scene") and switches to a 360-degree full-screen or rectangular display depending on the information from the emotion engine. Specifically, it analyzes the emotion data and dynamically changes the video display format. The output is video data adjusted to the optimal display format.

[0741] Step 6: Merging and saving the final footage

[0742] The server reintegrates all meshes after completion of the interpolation process and generates the final spherical projection video file (input: list of interpolated mesh images). It uses a video processing library (e.g., FFmpeg) to concatenate each frame and save it as a single continuous video file on the server. Specifically, it concatenates and encodes the frames and saves the result in an appropriate file format (e.g., converted_movie.mp4). The output is the saved spherical projection video file.

[0743] Step 7: Projecting the image

[0744] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium (input: final video file). A 360-degree compatible projector (e.g., a Fisheye projector) is used to enlarge and display the image. Specifically, the projector's control software is used to decode the video file and project it onto the screen. The output is a 360-degree image displayed on the spherical screen.

[0745] Step 8: Select and watch a scene

[0746] The user operates the interface to select a specific scene and view it in the planetarium (input: user selection). The emotion engine monitors the user's emotions in real time and switches between full-screen and rectangular display depending on the emotion. Specifically, the scene display is adjusted based on the user's operations and emotional data. The output is the visual experience the user views.

[0747] (Application example 2)

[0748] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0749] Conventional systems that project video data onto a spherical screen have the problem of being unable to display images adaptively according to the user's emotions. Furthermore, in industrial environments, there has been a lack of systems that dynamically highlight important information when operators monitor the status of machinery. This has led to problems such as increased operator stress and reduced work efficiency.

[0750] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0751] In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, a generation AI means for filling in gaps between each grid-shaped division, means for the user to select a specific scene, means for converting the video into a 360-degree full screen or rectangular display depending on the selected scene, means for integrating and saving the final video for spherical projection, means for projecting the video onto the spherical screen, means for recognizing the user's emotions using an emotion engine and adaptively displaying information accordingly, and means for displaying the video on a head-mounted display in an industrial environment and dynamically emphasizing important information. This enables adaptive video display according to the user's emotions, improving the efficiency of operator monitoring work in industrial environments and reducing stress.

[0752] The "means for acquiring designated video data" is a method for acquiring and reading a video data file designated in advance by the user or the system.

[0753] The "means for dividing video data into a grid" is a method for dividing acquired video data into a fixed grid pattern and storing the data as individual grids.

[0754] "Generative AI means" refers to artificial intelligence technology that analyzes the gaps between divided grid-like areas and fills them in based on surrounding pixel information.

[0755] The "means for the user to select a particular scene" is a method in which the user operates the interface to select a particular video scene that he or she wishes to view.

[0756] "Means for converting the video to a 360-degree full screen or rectangular display depending on the selected scene" refers to a method for converting the video so that the scene selected by the user is displayed in a 360-degree full screen or a rectangular display in the center.

[0757] The "means for integrating and saving the final spherical projection image" is a method for combining the generated image frames into a single continuous file and saving it in a format that can be projected onto a spherical screen.

[0758] "Means for projecting images onto a spherical screen" refers to a method using a projector or hardware for projecting a stored image file onto a spherical screen.

[0759] An "emotion engine" is a technology that analyzes a user's facial expressions and voice and recognizes their emotional state in real time.

[0760] "Means for adaptively displaying information" refers to a method for dynamically adjusting the information and images displayed based on the user's emotional state.

[0761] "Industrial environment" refers to an environment where industrial activities take place, such as a factory or manufacturing site.

[0762] A "head-mounted display" is a device worn on the user's head that displays images and information directly into the user's field of vision.

[0763] The "means for dynamically highlighting important information" is a method for detecting important information in an industrial environment in real time and highlighting it to the user.

[0764] This invention is a system that uses a head-mounted display (HMD) in an industrial environment to visualize images in a factory in 360 degrees and dynamically highlight information according to the operator's emotions. An embodiment of this system will be described in detail below.

[0765] Server Roles

[0766] The server performs the following main functions:

[0767] Acquiring video data

[0768] The server acquires and reads the specified video data file, using a video processing library such as OpenCV or FFmpeg.

[0769] Mesh division of video

[0770] The server divides the acquired video data into a grid. Specifically, it divides the video frame into 10x10 meshes and saves each mesh as separate data.

[0771] Interpolation between meshes

[0772] The server uses generative AI to fill in the gaps between the divided meshes. Using a generative AI model (such as GPT-3.5), it analyzes the boundaries of each mesh and generates pixels to fill in the gaps based on information from surrounding pixels. This filling process smooths the connections between the meshes, generating a continuous image suitable for 360-degree projection.

[0773] Emotion engine integration

[0774] The server is integrated with an emotion engine that recognizes the user's emotions. For example, using Azure Cognitive Services and TensorFlow, it analyzes changes in the user's facial expressions and voice to recognize emotions in real time. Based on this information, the video scene selection and display format are automatically adjusted.

[0775] Scene selection and processing

[0776] In addition to the user selecting a specific scene, the emotion engine monitors the user's emotions in real time, and based on this information, the video will either switch to a full 360-degree screen view or adjust to a central rectangular view depending on the emotion.

[0777] Merging and saving the final footage

[0778] The server then combines all the processed frames to generate the final 360-degree projection video file. The resulting video file is saved on the server in a format such as converted_movie.mp4. This is done by using a video processing library to connect the frames and output them as a single continuous video file.

[0779] Device Role

[0780] Image projection

[0781] The device receives the final video file sent from the server and displays it on the head-mounted display. The video is displayed in 360 degrees using the display device connected to the device. The video display format is automatically adjusted based on information from the emotion engine.

[0782] User Roles

[0783] Select a scene

[0784] Users can operate the interface to select specific scenes in the video. They can also check their own emotional state through the interface. The emotion engine analyzes the user's facial expressions and voice and provides the optimal video display format.

[0785] Viewing

[0786] The user wears a head-mounted display and watches a 360-degree visualization. The emotion engine recognizes emotions in real time and can highlight important information and issue warnings if the user is feeling stressed.

[0787] Specific examples

[0788] For example, if an operator is monitoring machinery in a factory and the emotion engine detects that the operator's stress level is high, the HMD image will instantly change based on that information. A 360-degree visualization of the machine that detected the abnormality will be displayed, highlighting the important parts. Below is an example of a prompt using a generative AI model:

[0789] "What is the best way to acquire real-time video data from machines in this factory and display it appropriately based on the operator's emotions?"

[0790] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0791] Step 1:

[0792] Acquiring video data

[0793] The server retrieves and reads the specified video data file. A video file name such as movie.mp4 is given as input data. The server reads this file using a video processing library such as OpenCV or FFmpeg and obtains the video frames. The output is the image data for each frame.

[0794] Step 2:

[0795] Grid division of video data

[0796] The server divides the captured video frame into a grid pattern. The input data is the video frame captured in the previous step. The video frame is divided into 10x10 meshes, and each mesh is saved as separate data. The output is the data for each mesh area.

[0797] Step 3:

[0798] Interpolation between meshes

[0799] The server uses a generative AI method to fill in the gaps between each divided mesh region. The input data is the data for the divided mesh regions. Using a generative AI model (e.g., GPT-3.5), it analyzes the boundaries of each mesh and generates pixels in the gaps based on the surrounding pixel information. The output is the mesh region data after the completion of the filling process.

[0800] Step 4:

[0801] Emotion recognition

[0802] The server uses an emotion engine to recognize the user's emotions in real time. The input data is the user's facial expression and voice data. The emotion engine (e.g., Azure Cognitive Services) analyzes the data and recognizes the user's emotional state. The output is the recognized emotional state data.

[0803] Step 5:

[0804] Select a scene

[0805] The user operates the interface to select a specific scene. The input data is a list of scenes. For each scene selected by the user, the server identifies the video data for that scene. The output is the video data for the selected scene.

[0806] Step 6:

[0807] Video display format conversion

[0808] The server converts the video into a 360-degree full-screen or central rectangular display depending on the selected scene. The input data is the video data of the selected scene and the emotional state data recognized in the previous step. The server converts the video into the optimal display format based on the emotional state. The output is the video data in the converted display format.

[0809] Step 7:

[0810] Merging and saving the final footage

[0811] The server combines and saves the converted video frames as a single continuous file. The input data is the converted video frame data. Using a video processing library, it concatenates these frames and saves them in a format such as converted_movie.mp4. The output is the final video file.

[0812] Step 8:

[0813] Image projection

[0814] The terminal receives the final video file sent from the server and displays it on the head-mounted display. The input data is the saved final video file. This data is sent to the HMD, which displays a 360-degree visualized video. The output is the video displayed on the HMD.

[0815] Step 9:

[0816] User reviews

[0817] The user wears a head-mounted display and watches a dynamically adjusted 360-degree video. The input data is the video displayed on the HMD. The output is the user's emotional response, which is then fed back to the emotion engine.

[0818] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0819] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0820] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0821] [Third embodiment]

[0822] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0823] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0824] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0825] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0826] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0827] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0828] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0829] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0830] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0831] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0832] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0833] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0834] This invention is a system that uses generative AI to convert existing video data into images that can be projected onto a 360-degree spherical screen. This system is composed of a server, terminals, and users working together, and each step plays a different role.

[0835] Server Roles

[0836] Acquiring video data

[0837] The server retrieves the specified video data file. For example, if a video file named movie.mp4 is specified, the server reads it and starts processing.

[0838] Mesh division of video

[0839] The server divides the acquired video data into a grid. For example, it divides a video frame into a 10x10 mesh and saves each mesh in a format that can be processed individually.

[0840] Interpolation between meshes

[0841] The server uses generation AI to fill in the gaps between the divided meshes. The generation AI analyzes the boundaries of each mesh and generates a complementary image based on information from surrounding pixels. This results in a smooth transition between images, making it suitable for spherical projection.

[0842] Scene selection and processing

[0843] Through the interface, the user selects whether to display a particular scene (e.g., ocean or space) in full 360-degree screen or in a central rectangular view. This selection information is sent to the server, which then performs the appropriate processing for each scene.

[0844] Merging and saving the final footage

[0845] The server merges all the processed frames and generates the final spherical projection video file, which is saved on the server in the format converted_movie.mp4, for example.

[0846] Device Role

[0847] Image projection

[0848] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. Using a projector connected to the terminal, the video is expanded and displayed on a 360-degree spherical screen, allowing the audience to experience the video in all directions.

[0849] User Roles

[0850] Select a scene

[0851] The user selects a specific scene from the video through an interface (e.g., dedicated software or a web application). Based on this selection, the system determines whether that scene should be displayed as a full 360-degree screen, while the other scenes should be displayed as a rectangular area in the center.

[0852] Viewing

[0853] Users can view images projected onto a spherical screen in the planetarium. The immersive feeling provided by 360-degree images allows users to enjoy a new visual experience that is different from that of a conventional flat screen.

[0854] Specific examples

[0855] For example, if a user wants to view an "ocean" scene in full 360-degree screen, the user selects the "ocean" scene through the interface. Based on this information, the server uses AI to generate the "ocean" scene for full 360-degree screen viewing, while other scenes (e.g., indoor scenes) are displayed as rectangular images in the center. Finally, the generated spherical projection video file is sent from the server to the device and projected onto the spherical screen of the planetarium. This process allows the user to enjoy a new visual experience.

[0856] Through these processing steps, the system of the present invention can easily convert existing images for spherical projection, providing a highly immersive experience.

[0857] The processing flow will be explained below.

[0858] Step 1: Acquire video data

[0859] The server retrieves the specified video data file. For example, if the file name is movie.mp4, the server reads this file and starts processing it. The video file is split into frames using a video processing library, and each frame is saved in a buffer.

[0860] Step 2: Meshing the image

[0861] The server divides each loaded frame into a grid. Specifically, it divides the frame into 10x10 meshes and treats each mesh as individual data. This process is performed using an image processing library.

[0862] Step 3: Interpolation between meshes

[0863] The server uses generative AI to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates pixels to fill in the gaps based on information from surrounding pixels. This filling process smooths the connections between the meshes, generating a continuous image suitable for spherical projection.

[0864] Step 4: Select a Scene

[0865] The user operates an interface to select specific scenes in the video, for example, using dedicated software or a web application. For each scene, the user can choose between a full-screen 360-degree view or a central rectangular view.

[0866] Step 5: Decide on the scene display format

[0867] The server receives the scene information selected by the user and determines the display format for each scene based on that information: certain scenes are processed for a full 360-degree screen display, while others are processed for a central rectangular display.

[0868] Step 6: Merging and saving the final footage

[0869] The server then combines all the processed frames to generate the final spherical projection video file. The resulting video file is saved on the server in the format converted_movie.mp4, for example. To do this, a video processing library is used to connect the frames and output them as a single continuous video file.

[0870] Step 7: Projecting the image

[0871] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. Using a projector connected to the terminal, the video is enlarged and displayed on a 360-degree spherical screen. This projection allows the audience to enjoy the video in all directions.

[0872] Step 8: Watch the video

[0873] Users can view images projected onto a spherical screen in the planetarium. The 360-degree images provide a new visual experience that differs from that of conventional flat screens.

[0874] Example 1

[0875] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0876] Conventional video projection technology has difficulty adapting existing flat images to a 360-degree spherical screen, making it difficult to maintain the continuity and smoothness of the image. Furthermore, there are limited ways for users to select specific scenes and determine the optimal display method. This makes it difficult to provide a highly immersive visual experience.

[0877] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0878] In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, a generating AI means for filling in the gaps between each grid-shaped division, means for the user to select a specific scene, means for converting the video into a 360-degree full screen or a central rectangular display depending on the selected scene, means for integrating and saving the final video for spherical projection, and means for projecting the integrated video onto a spherical screen. This makes it possible to smoothly convert existing video for spherical projection and provide the optimal display method based on the user's selection.

[0879] The "means for acquiring the specified video data" refers to the means used by the server to read and process the video file specified by the user.

[0880] The "means for dividing image data into a grid" is a means for dividing acquired image data based on a certain mesh size, making it easier to process the data individually.

[0881] The "generative AI means for filling in the gaps between each grid-like divided area" is a means for using a generative AI model to fill in the gaps that occur between divided meshes, thereby maintaining the continuity and smoothness of the image.

[0882] The "means for the user to select a particular scene" refers to an interface or operation means for the user to select a particular scene or location that he or she wishes to display.

[0883] The "means for converting the video into a 360-degree full screen or a rectangular display in the center according to the selected scene" is a means for converting the display format of the video into a 360-degree full screen or a rectangular display in the center based on the user's selection.

[0884] The "means for integrating and storing final spherical projection images" refers to a means for integrating interpolated and transformed image frames and storing them in a format suitable for a spherical screen.

[0885] The "means for projecting the integrated image onto a spherical screen" refers to a means for projecting the integrated and stored image onto a spherical screen using a projection device.

[0886] This invention is a system that uses a generative AI model to convert existing video data into images that can be projected onto a 360-degree spherical screen. This system is composed of a server, a terminal, and a user working together, and each step plays a different role.

[0887] Server Roles

[0888] Acquiring video data

[0889] The server retrieves the video data file specified by the user. For example, if the user specifies a file called "movie.mp4" through the interface, the server reads this file from the file system and saves it in a temporary directory.

[0890] Mesh division of video

[0891] The server divides the acquired video data into 10x10 meshes for each frame. This process uses an image processing library (e.g., OpenCV). The divided mesh parts are saved in a format that can be processed individually.

[0892] Interpolation between meshes

[0893] The server uses a generative AI model (e.g., GAN or VQ-VAE) to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates a complementary image based on information from surrounding pixels. This results in a smooth transition between the images, making them suitable for spherical projection.

[0894] Scene selection and processing

[0895] The user uses an interface (dedicated software or web application) to select the specific scene they want to view. For example, if the user wants to view an "ocean" scene in full 360 degrees, the server receives this information and generates a video centered on that scene.

[0896] Merging and saving the final footage

[0897] The server merges all processed frames and generates the final spherical projection video file, which is saved in MP4 format and named "converted_movie.mp4" for example.

[0898] Device Role

[0899] Image projection

[0900] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. A projection device (e.g., a 4K projector) connected to the terminal projects the video onto the 360-degree spherical screen, allowing the audience to experience the video in all directions.

[0901] User Roles

[0902] Select a scene

[0903] Through the interface, the user selects a particular scene in the video, and based on this selection, the server performs the appropriate processing for that scene and generates the video.

[0904] Viewing

[0905] Users can view images projected onto a spherical screen in the planetarium. The immersive feeling provided by 360-degree images allows users to enjoy a new visual experience that is different from that of a conventional flat screen.

[0906] Examples and prompts

[0907] For example, if a user wants to use an interface to view a full-screen 360-degree "ocean" scene, they can use the following prompt:

[0908] "Create a full-screen 360-degree ocean scene."

[0909] Based on this prompt, the server uses AI to generate the "ocean" scene, and other scenes (such as indoor scenes) are displayed as a central rectangular image. The generated spherical projection image file is sent from the server to the terminal, which then projects the image onto the spherical screen of the planetarium.

[0910] Through these processing steps, the system of the present invention can easily convert existing images for spherical projection, providing a highly immersive experience.

[0911] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0912] Program processing flow

[0913] Step 1: Acquire video data

[0914] Specific behavior:

[0915] The server reads the video file (e.g., "movie.mp4") specified by the user through the interface. It checks whether the file exists and whether read permissions are correct. The video data is saved in a temporary directory.

[0916] Input and Output:

[0917] Input: User-specified video file name ("movie.mp4")

[0918] Data processing and data calculation: Checking file existence and reading

[0919] Output: Video data is saved to a temporary directory

[0920] Step 2: Mesh division of video data

[0921] Specific behavior:

[0922] The server divides the acquired video data into 10x10 meshes for each frame. This process uses an image processing library (e.g., OpenCV). The divided mesh parts are saved in a format that can be processed individually.

[0923] Input and Output:

[0924] Input: Video data saved in the temporary directory

[0925] Data processing and data calculation: Divide the video frame into 10x10 meshes

[0926] Output: The divided mesh parts are saved as temporary files.

[0927] Step 3: Interpolation between meshes

[0928] Specific behavior:

[0929] The server uses a generative AI model (e.g., GAN or VQ-VAE) to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates a fill-in image based on the surrounding pixel information.

[0930] Input and Output:

[0931] Input: Divided mesh parts

[0932] Data processing and data calculation: Fill gaps between meshes with generated AI

[0933] Output: The completed mesh part is resaved

[0934] Step 4: Select and process the scene

[0935] Specific behavior:

[0936] The user selects the specific scene they want to display using an interface (dedicated software or web application). The server receives this information and generates a video centered on that scene. For example, if you want to display an "ocean" scene in full 360 degrees, the generation AI will perform the interpolation process.

[0937] Input and Output:

[0938] Input: User's scene selection information (e.g., "ocean" scene)

[0939] Data processing and data calculation: Generate images based on selected scenes and process them with AI

[0940] Output: Video data of the processed scene

[0941] Step 5: Merging and saving the final footage

[0942] Specific behavior:

[0943] The server will then combine all the interpolated and processed frames to generate the final spherical projection video file, which will be saved in MP4 format and named "converted_movie.mp4" for example.

[0944] Input and Output:

[0945] Input: Interpolated and processed frames

[0946] Data processing and data calculation: Frame integration and video file generation

[0947] Output: Final spherical projection video file

[0948] Step 6: Projecting the image

[0949] Specific behavior:

[0950] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium using a projection device (e.g., a 4K projector) connected to the terminal.

[0951] Input and Output:

[0952] Input: Final spherical projection image file

[0953] Data processing and data calculation: Playback of video files

[0954] Output: Image projected onto a spherical screen

[0955] Step 7: Select a Scene

[0956] Specific behavior:

[0957] The user selects a particular scene through the interface, and when the user selects a scene and clicks the "View" button on the interface, this information is sent to the server, which processes the scene.

[0958] Input and Output:

[0959] Input: User scene selection information

[0960] Data Processing and Data Computation: Scene-Based Video Processing

[0961] Output: Footage of the processed scene

[0962] Step 8: Viewing

[0963] Specific behavior:

[0964] Users can enjoy the 360-degree immersive experience by viewing images projected onto a spherical screen in a planetarium.

[0965] Input and Output:

[0966] Input: Image projected onto a spherical screen

[0967] Data processing and calculation: None

[0968] Output: The immersive visual experience the user experiences

[0969] In this way, the system performs appropriate processing at each step to adapt existing footage to a 360-degree spherical screen, providing users with a new visual experience.

[0970] (Application example 1)

[0971] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0972] Conventional video streaming services require special equipment and complex operations to enhance the user's sense of immersion, often limiting the quality of the visual experience. Furthermore, there is a need for an efficient method to convert existing video data into 360-degree spherical video and enable real-time viewing.

[0973] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0974] In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, a generation AI means for filling in gaps between each grid-shaped division, means for the user to select a specific scene, means for converting the video into a 360-degree full screen or rectangular display depending on the selected scene, means for integrating and saving the final video for spherical projection, means for streaming the final video for spherical projection to the user's terminal, and means for performing video data interpolation processing based on a 10x10 mesh. This allows users to convert existing video data into high-quality 360-degree spherical video and enjoy an immersive visual experience in real time without the need for special equipment or complex operations.

[0975] The "means for acquiring the designated video data" is a means by which the server receives the video data file that the user wants to view and starts processing.

[0976] The "means for dividing image data into a grid" is a means for dividing the acquired image data into a grid of a certain size and saving each mesh in a processable form.

[0977] The "generative AI means for filling in the gaps between each grid-like divided area" is a means for using a generative AI to analyze the gaps between divided meshes and generate a filled image based on surrounding pixel information.

[0978] The "means for the user to select a particular scene" is a means for the user to select a particular scene that he or she wishes to display through an interface.

[0979] "Means for converting the video into a 360-degree full screen or rectangular display according to the selected scene" refers to means for converting the video into a 360-degree full screen display or a central rectangular display based on the selected scene.

[0980] The "means for integrating and saving the final spherical projection image" is a means for integrating all the processed frames and saving them on a server as a final spherical projection image file.

[0981] The "means for projecting an image onto a spherical screen" refers to a means for projecting the final image received by the terminal onto a spherical screen, thereby displaying an image that spreads in all directions.

[0982] The "means for streaming the final spherical projection image to the user's terminal" refers to a means for transmitting the final image generated by the server to the user's terminal in real time, enabling continuous playback.

[0983] "Means for performing video data interpolation processing based on a 10x10 mesh" refers to a means for dividing a video frame into a 10x10 mesh and using a generation AI to interpolate the gaps between the meshes.

[0984] This invention is a system that uses generative AI to convert existing video data into video that can be projected onto a 360-degree spherical screen, enabling real-time viewing on a smartphone. The system is composed of a server, terminals, and users working together.

[0985] Server Roles

[0986] Acquiring video data

[0987] The server retrieves the specified video data file. For example, when a user specifies the file name of a movie or video they want to watch, the server reads the video data and begins processing.

[0988] Mesh division of video

[0989] The server divides the acquired video data into a 10x10 grid. Each mesh is saved in a format that can be processed individually. This allows subsequent processing by the generation AI to be performed efficiently.

[0990] Interpolation between meshes

[0991] The server uses a generative AI model to fill in the gaps between the divided meshes. The generative AI analyzes the boundaries of each mesh and generates a complementary image based on surrounding pixel information. This results in a smooth transition between images, making them suitable for spherical projection.

[0992] Scene selection and processing

[0993] The user selects a specific scene (e.g., ocean or space) through the interface. Based on this selection, the server displays the specific scene as a full 360-degree screen and processes the other scenes as a rectangular display in the center.

[0994] Merging and saving the final footage

[0995] The server then combines all the processed frames to generate the final spherical projection video file, which is then stored on the server and subsequently streamed to the user's device in real time.

[0996] Device Role

[0997] Projecting and viewing images

[0998] The device receives the final video file sent from the server and displays it as a 360-degree spherical image, allowing users to view images in all directions using their smartphones. This provides a high-quality immersive experience without the need for special equipment.

[0999] User Roles

[1000] Select a scene

[1001] Users select the specific scenes they want to watch through dedicated software or a web application, and the server performs the necessary video processing based on this selection.

[1002] Watching videos

[1003] Users can watch 360-degree spherical images streamed in real time on their smartphones, providing a new visual experience that is different from that of conventional flat screens.

[1004] Hardware and software used

[1005] Hardware: Servers, smartphones

[1006] Software: Python, OpenCV, MoviePy, FastAI

[1007] Data processing and calculation

[1008] Data Acquisition and Meshing

[1009] Using Python and OpenCV, we take the video data and divide it into a 10x10 mesh.

[1010] Interpolation between meshes

[1011] Using FastAI, the generative AI model complements the mesh boundaries of the video frame to generate smooth images.

[1012] Video integration and storage

[1013] The final video is integrated and saved using OpenCV and MoviePy, and data for streaming distribution is generated.

[1014] Examples and prompts

[1015] As a concrete example, let's say a user wants to display an "ocean" scene in full 360-degree screen. The user selects the "ocean" scene using the interface. Based on this information, the server uses AI to generate the "ocean" scene for full 360-degree screen display, and processes other scenes (e.g., indoor scenes) as rectangular displays in the center.

[1016] Prompt Sentence Examples

[1017] "Divide the following video data into a 10x10 mesh, and use a generative AI model to fill in the gaps between the meshes to generate an image for 360-degree spherical projection."

[1018] This system allows users to easily enjoy high-quality 360-degree spherical images.

[1019] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1020] Step 1:

[1021] Acquiring video data

[1022] The server retrieves the video data file (e.g., movie.mp4) specified by the user. Specifically, the user specifies the video file they want to watch, and the server reads that file. In this process, the path and name of the video file are input data, and the retrieved video is output.

[1023] Step 2:

[1024] Mesh division of video

[1025] The server divides the acquired video data into a grid. Specifically, it uses Python's OpenCV library to divide the video frame into a 10x10 mesh and saves each mesh in a processable format. The input of this step is the acquired video frame, and the output is each divided mesh.

[1026] Step 3:

[1027] Interpolation between meshes

[1028] The server uses a generative AI model to fill in the gaps between the meshes. Specifically, it uses the FastAI library to have the generative AI model analyze the boundaries of each mesh and generate a filled image based on the surrounding pixel information. The input to this step is the segmented mesh data, and the output is the filled image data.

[1029] Step 4:

[1030] Select a scene

[1031] The user selects a specific scene through the interface. Specifically, the user selects the scene they want to display (e.g., "ocean" or "space") using dedicated software or a web application, and this information is sent to the server. The input of this step is the user's selection information, and the output is the selected scene information.

[1032] Step 5:

[1033] Scene-specific video processing

[1034] The server processes the video according to the selected scene. Specifically, it converts certain scenes into a 360-degree full-screen display based on the selection information, and other scenes into a rectangular display with the selection information at the center. The video conversion is performed using Python and OpenCV. The input of this step is the selected scene information and the interpolated video data, and the output is the converted video data.

[1035] Step 6:

[1036] Merging and saving the final footage

[1037] The server merges all processed video frames to generate the final spherical projection video file. Specifically, it uses OpenCV and MoviePy to merge the frames into a single video file. The input of this step is the converted video data, and the output is the merged final video file.

[1038] Step 7:

[1039] Video streaming

[1040] The server then streams the final video file to the user's device. Specifically, it uses HTTP streaming technology to send the video to the user's smartphone in real time. The input of this step is the final video file, and the output is the video played on the user's device.

[1041] Step 8:

[1042] Projecting and viewing images

[1043] The user's device displays the final video received from the server as a 360-degree spherical video. Specifically, a smartphone application plays the video, allowing the user to view the omnidirectional video without the need for special equipment. The input of this step is the streamed video data, and the output is the user's visual experience.

[1044] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1045] This invention is a system that uses generative AI to convert existing video data into video that can be projected onto a 360-degree spherical screen, and combines it with an emotion engine to realize a video display that responds to the user's emotions. This system is composed of a server, terminals, and users working together, and each step plays a different role.

[1046] Server Roles

[1047] Acquiring video data

[1048] The server retrieves and reads the specified video data file. For example, if the file name is movie.mp4, the server reads this file and starts processing.

[1049] Mesh division of video

[1050] The server divides the acquired video data into a grid. Specifically, it divides the video frame into 10x10 meshes and saves each mesh as separate data.

[1051] Interpolation between meshes

[1052] The server uses generative AI to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates pixels to fill in the gaps based on information from surrounding pixels. This filling process smooths the connections between the meshes, generating a continuous image suitable for spherical projection.

[1053] Emotion engine integration

[1054] The server is integrated with an emotion engine that recognizes the user's emotions. This emotion engine analyzes changes in the user's facial expressions and voice to recognize emotions in real time, and automatically adjusts the video scene selection and display format based on that information.

[1055] Scene selection and processing

[1056] In addition to the user selecting a specific scene, the emotion engine monitors the user's emotions in real time, and based on this information, the video will either switch to a full 360-degree screen view or adjust to a central rectangular view depending on the emotion.

[1057] Merging and saving the final footage

[1058] The server then combines all the processed frames to generate the final spherical projection video file. The resulting video file is saved on the server in the format converted_movie.mp4, for example. To do this, a video processing library is used to connect the frames and output them as a single continuous video file.

[1059] Device Role

[1060] Image projection

[1061] The device receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. The video is enlarged and displayed on a 360-degree spherical screen using a projector connected to the device. The video display format is automatically adjusted based on information from the emotion engine.

[1062] User Roles

[1063] Select a scene

[1064] Users can operate the interface to select specific scenes in the video. They can also check their own emotional state through the interface. The emotion engine analyzes the user's facial expressions and voice and provides the optimal video display format.

[1065] Viewing

[1066] Users view images projected onto a spherical screen in the planetarium. The emotion engine recognizes emotions in real time, and if the user is feeling emotional, it switches to a full-screen 360-degree display of a specific scene. In this way, users can enjoy a highly immersive visual experience.

[1067] Specific examples

[1068] For example, if a user is viewing a 360-degree full-screen "ocean" scene in a planetarium, the emotion engine will analyze the user's facial expressions and, if it detects signs of emotion, automatically switch to a full-screen 360-degree display of the next scene. If the emotion engine's analysis determines that the user is bored, it will switch to a rectangular display in the center and provide other content. This process ensures that the user always receives the optimal visual experience.

[1069] Through these processing steps, the system of the present invention can easily convert existing images for spherical projection, providing a highly immersive experience. Furthermore, by integrating an emotion engine, it is possible to realize image display that adapts to the user's emotions.

[1070] The processing flow will be explained below.

[1071] Step 1: Acquire video data

[1072] The server retrieves the specified video data file. Specifically, if the file name is movie.mp4, the server reads this file and uses a video processing library (e.g., OpenCV) to store each frame in a buffer.

[1073] Step 2: Meshing the image

[1074] The server divides each loaded frame into a grid. Specifically, it divides each frame into 10x10 meshes and saves each mesh as separate data. This separates the fine details of the image, making it easier to perform subsequent interpolation processing.

[1075] Step 3: Interpolation between meshes

[1076] The server uses generative AI to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates pixels to fill in the gaps based on information from surrounding pixels. This step smooths the connections between the meshes, resulting in a continuous image suitable for spherical projection.

[1077] Step 4: Obtaining emotion data

[1078] The emotion engine integrated into the server acquires the user's facial expression and voice data to recognize the user's emotions in real time. This involves capturing the user's facial expressions and voice using a camera and microphone, and analyzing the emotions using emotion recognition algorithms.

[1079] Step 5: Select a Scene

[1080] The user operates the interface to select a specific scene. When the user selects a scene, the emotion engine's analysis results are also taken into account. For example, if the user is emotional, the emotion engine will automatically suggest an enhanced scene display.

[1081] Step 6: Decide on the scene display format

[1082] The server determines the display format for each scene based on the scene information selected by the user and the analysis results of the emotion engine. Specifically, if the server recognizes that the user is moved, it displays a specific scene in full 360-degree screen, spreading the image in all directions instead of displaying it in a rectangular format in the center.

[1083] Step 7: Merging and saving the final footage

[1084] The server then combines all the processed frames to generate the final spherical projection video file (e.g., converted_movie.mp4). The resulting video file is then concatenated using a video processing library to create a single continuous video file, which is then saved on the server.

[1085] Step 8: Projecting the image

[1086] The device receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. Using a projector connected to the device, the video is expanded and displayed on a 360-degree spherical screen. The video is optimally adjusted based on the analysis of the emotion engine.

[1087] Step 9: Watch the video

[1088] Users watch images projected onto a spherical screen in the planetarium. The emotion engine analyzes the user's emotions in real time, adjusting the projection of specific scenes if the user is excited, for example, to optimize the user experience. In this way, users can enjoy a highly immersive visual experience.

[1089] Example 2

[1090] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1091] Conventional video display systems have had difficulty optimizing video display in response to user emotions. Furthermore, complex data processing and advanced technology are required to project existing video data in a form suitable for a spherical screen. This can result in a degradation of the quality of the user experience. The purpose of this invention is to solve these problems and realize a highly immersive video display that responds to the user's emotions.

[1092] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, and a generation AI means for filling in gaps between each grid-shaped division. This makes it possible to process existing video data into an image suitable for spherical projection. Furthermore, by including means for a user to select a specific scene, means for converting the image into a 360-degree full screen or rectangular display depending on the selected scene, means for integrating and saving the final image for spherical projection, and means for projecting the image onto the spherical screen, it is possible to realize an image display that is optimal for the scene and emotion specified by the user. Furthermore, by including emotion engine means for recognizing the user's emotion and means for automatically adjusting the image display format according to the user's emotion using the emotion engine means, it is possible to realize real-time adjustment of the image display according to the user's emotion.

[1093] Definitions of important words

[1094] "Specified video data" refers to a video or video file specified by a user or the system prior to entering the system.

[1095] "Generative AI means" refers to a method or device that uses artificial intelligence technology to fill in gaps or missing parts of video data and generate smooth, continuous video.

[1096] "User selection means" refers to a method or device that allows a user to select a particular scene or video portion from the interface.

[1097] "Emotion engine means" refers to technology or a device that analyzes a user's facial expressions, voice, and other biometric information to recognize the user's emotional state in real time.

[1098] "Means for transforming video into a 360-degree full-screen or rectangular display" refers to technology or devices that adapt video data to an all-around display format or a specific shape based on the user's visual preferences.

[1099] "Means for integrating and saving the final spherical projection image" refers to the technology and equipment for integrating and saving the results of each process into a single continuous image file.

[1100] "Means for projecting an image onto a spherical screen" refers to a method or device for displaying the generated spherical projection image on a screen capable of displaying 360 degrees.

[1101] "Means for automatically adjusting the video display format according to the user's emotions using emotion engine means" refers to technology or devices that dynamically change the video display format based on the results of user emotion recognition, providing an optimal visual experience.

[1102] MODE FOR CARRYING OUT THE INVENTION

[1103] This invention is a system that uses a generative AI model to convert existing video data into video that can be projected onto a 360-degree spherical screen, and combines it with an emotion engine to realize video display that responds to the user's emotions. This system is composed of a server, terminals, and users working together, and the roles of each are explained in detail below.

[1104] Server Roles

[1105] Acquiring video data

[1106] The server retrieves and reads the specified video data file. Specifically, if the user specifies a file called "movie.mp4", the server reads the file from a specified directory and loads it into memory as byte data. This retrieval process uses standard file I / O operations and byte stream processing libraries (e.g., Python's os module and io module).

[1107] Mesh division of video

[1108] The server divides the acquired video data into a grid for each frame. Specifically, it uses an image processing library (e.g., OpenCV library) to divide each frame into a 10x10 mesh. Each mesh portion is temporarily saved as individual small image data. This allows the next interpolation process to proceed smoothly.

[1109] Interpolation between meshes

[1110] The server uses a generative AI model to fill in the gaps between each divided mesh. To do this, the server uses a generative AI model (e.g., GANs or AutoEncoder) to analyze surrounding pixel information and generate pixels to fill in the gaps. Specifically, the server trains the model using a deep learning framework such as PyTorch or TensorFlow, and outputs the filled-in parts based on the boundary pixel information of each mesh. This filling process smooths the connections between the meshes.

[1111] Emotion engine integration

[1112] The server is integrated with an emotion engine that analyzes facial and voice data sent from the user's webcam and microphone. This analysis uses face recognition APIs and voice analysis libraries (e.g., OpenCV, Dlib, and Google's Cloud Speech-to-Text API). The analyzed emotion information is used to adjust the video display format.

[1113] Scene selection and processing

[1114] The server selects an appropriate video scene based on the user's interface operations and data from the emotion engine. For example, if the user selects an "ocean scene" on the interface, the server obtains frame data for that scene, and if the emotion engine detects the user's emotion, it changes the video to 360-degree full-screen mode.

[1115] Merging and saving the final footage

[1116] The server reintegrates all meshes after the interpolation process is completed to generate the final spherical projection video file. It then uses a video processing library (e.g., FFmpeg) to concatenate each frame and save it on the server as a single continuous video file (e.g., converted_movie.mp4). This process ensures that all frames are displayed continuously.

[1117] Device Role

[1118] Image projection

[1119] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. The image is enlarged using a 360-degree projector (e.g., a fisheye projector) connected to the terminal. The display format of the image is automatically adjusted according to the user's emotions, providing the user with an enhanced visual experience.

[1120] User Roles

[1121] Select a scene

[1122] The user operates the interface to select a specific scene. The user can also check their own emotional state through the interface. The emotion engine analyzes the user's facial expressions and voice and provides a video display format that best suits the user's emotions.

[1123] Viewing

[1124] Users view images projected onto a spherical screen in a planetarium. The emotion engine monitors the user's emotions in real time, and if the user is emotionally moved, the scene switches to a full 360-degree screen. This process allows users to view the images with a high level of immersion.

[1125] Examples and prompts

[1126] For example, consider a case where a user is viewing an "ocean scene" in a planetarium while viewing it in full 360-degree screen. If the emotion engine analyzes the user's facial expression and recognizes signs of emotion, it automatically switches to displaying the next scene in full 360-degree screen. Conversely, if it determines that the user is bored, it switches the image to a rectangular display in the center and provides other content.

[1127] An example prompt is, "Convert the movie file movie.mp4 into a 360-degree video and use the emotion engine to adjust the display format according to the user's emotion."

[1128] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1129] Program processing flow

[1130] Step 1: Acquire video data

[1131] The server retrieves the specified video data file (input: specified file path). Based on the prompt, it retrieves the specified file (e.g., movie.mp4) from the directory and loads it into memory as byte data. Specifically, it opens the file using Python's os and io modules and reads its contents as a data stream. The output is the video data loaded into memory.

[1132] Step 2: Meshing the image

[1133] The server divides the acquired video data into a grid for each frame (input: video data in memory). Specifically, it uses an image processing library (e.g., OpenCV) to divide each frame into a 10x10 mesh. Each mesh portion is temporarily saved as individual image data. Specifically, it divides the frame into an array, and then divides it into a grid using a cv2 function. The output is a list of each divided mesh image.

[1134] Step 3: Interpolation between meshes

[1135] The server uses a generative AI model to fill in the gaps between each divided mesh (input: list of divided mesh images). Using a generative AI model (e.g., GANs or AutoEncoder), it analyzes surrounding pixel information and generates pixels in the gaps. Specifically, it uses PyTorch or TensorFlow to input the boundary data of each mesh into the AI ​​model, which then outputs the filled pixel information. The output is a continuous mesh image with the gaps filled in.

[1136] Step 4: Integrating the Emotion Engine

[1137] The server integrates an emotion engine that analyzes the user's emotions in real time (input: user's facial expression data and voice data). The emotion engine uses face recognition APIs and voice analysis libraries (e.g., OpenCV and Dlib, Google Cloud Speech-to-Text API) to analyze the user's emotional state. The emotion data is used to adjust the video display. Specifically, it analyzes real-time streams from the webcam and microphone and outputs the type and intensity of emotions as data. The output is the analyzed emotion information.

[1138] Step 5: Select and process the scene

[1139] The server selects and processes video scenes based on user operations and data from the emotion engine (input: user-selected data and emotion information). Specifically, it obtains the scene selected in the user interface (e.g., "ocean scene") and switches to a 360-degree full-screen or rectangular display depending on the information from the emotion engine. Specifically, it analyzes the emotion data and dynamically changes the video display format. The output is video data adjusted to the optimal display format.

[1140] Step 6: Merging and saving the final footage

[1141] The server reintegrates all meshes after completion of the interpolation process and generates the final spherical projection video file (input: list of interpolated mesh images). It uses a video processing library (e.g., FFmpeg) to concatenate each frame and save it as a single continuous video file on the server. Specifically, it concatenates and encodes the frames and saves the result in an appropriate file format (e.g., converted_movie.mp4). The output is the saved spherical projection video file.

[1142] Step 7: Projecting the image

[1143] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium (input: final video file). A 360-degree compatible projector (e.g., a Fisheye projector) is used to enlarge and display the image. Specifically, the projector's control software is used to decode the video file and project it onto the screen. The output is a 360-degree image displayed on the spherical screen.

[1144] Step 8: Select and watch a scene

[1145] The user operates the interface to select a specific scene and view it in the planetarium (input: user selection). The emotion engine monitors the user's emotions in real time and switches between full-screen and rectangular display depending on the emotion. Specifically, the scene display is adjusted based on the user's operations and emotional data. The output is the visual experience the user views.

[1146] (Application example 2)

[1147] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1148] Conventional systems that project video data onto a spherical screen have the problem of being unable to display images adaptively according to the user's emotions. Furthermore, in industrial environments, there has been a lack of systems that dynamically highlight important information when operators monitor the status of machinery. This has led to problems such as increased operator stress and reduced work efficiency.

[1149] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1150] In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, a generation AI means for filling in gaps between each grid-shaped division, means for the user to select a specific scene, means for converting the video into a 360-degree full screen or rectangular display depending on the selected scene, means for integrating and saving the final video for spherical projection, means for projecting the video onto the spherical screen, means for recognizing the user's emotions using an emotion engine and adaptively displaying information accordingly, and means for displaying the video on a head-mounted display in an industrial environment and dynamically emphasizing important information. This enables adaptive video display according to the user's emotions, improving the efficiency of operator monitoring work in industrial environments and reducing stress.

[1151] The "means for acquiring designated video data" is a method for acquiring and reading a video data file designated in advance by the user or the system.

[1152] The "means for dividing video data into a grid" is a method for dividing acquired video data into a fixed grid pattern and storing the data as individual grids.

[1153] "Generative AI means" refers to artificial intelligence technology that analyzes the gaps between divided grid-like areas and fills them in based on surrounding pixel information.

[1154] The "means for the user to select a particular scene" is a method in which the user operates the interface to select a particular video scene that he or she wishes to view.

[1155] "Means for converting the video to a 360-degree full screen or rectangular display depending on the selected scene" refers to a method for converting the video so that the scene selected by the user is displayed in a 360-degree full screen or a rectangular display in the center.

[1156] The "means for integrating and saving the final spherical projection image" is a method for combining the generated image frames into a single continuous file and saving it in a format that can be projected onto a spherical screen.

[1157] "Means for projecting images onto a spherical screen" refers to a method using a projector or hardware for projecting a stored image file onto a spherical screen.

[1158] An "emotion engine" is a technology that analyzes a user's facial expressions and voice and recognizes their emotional state in real time.

[1159] "Means for adaptively displaying information" refers to a method for dynamically adjusting the information and images displayed based on the user's emotional state.

[1160] "Industrial environment" refers to an environment where industrial activities take place, such as a factory or manufacturing site.

[1161] A "head-mounted display" is a device worn on the user's head that displays images and information directly into the user's field of vision.

[1162] The "means for dynamically highlighting important information" is a method for detecting important information in an industrial environment in real time and highlighting it to the user.

[1163] This invention is a system that uses a head-mounted display (HMD) in an industrial environment to visualize images in a factory in 360 degrees and dynamically highlight information according to the operator's emotions. An embodiment of this system will be described in detail below.

[1164] Server Roles

[1165] The server performs the following main functions:

[1166] Acquiring video data

[1167] The server acquires and reads the specified video data file, using a video processing library such as OpenCV or FFmpeg.

[1168] Mesh division of video

[1169] The server divides the acquired video data into a grid. Specifically, it divides the video frame into 10x10 meshes and saves each mesh as separate data.

[1170] Interpolation between meshes

[1171] The server uses generative AI to fill in the gaps between the divided meshes. Using a generative AI model (such as GPT-3.5), it analyzes the boundaries of each mesh and generates pixels to fill in the gaps based on information from surrounding pixels. This filling process smooths the connections between the meshes, generating a continuous image suitable for 360-degree projection.

[1172] Emotion engine integration

[1173] The server is integrated with an emotion engine that recognizes the user's emotions. For example, using Azure Cognitive Services and TensorFlow, it analyzes changes in the user's facial expressions and voice to recognize emotions in real time. Based on this information, the video scene selection and display format are automatically adjusted.

[1174] Scene selection and processing

[1175] In addition to the user selecting a specific scene, the emotion engine monitors the user's emotions in real time, and based on this information, the video will either switch to a full 360-degree screen view or adjust to a central rectangular view depending on the emotion.

[1176] Merging and saving the final footage

[1177] The server then combines all the processed frames to generate the final 360-degree projection video file. The resulting video file is saved on the server in a format such as converted_movie.mp4. This is done by using a video processing library to connect the frames and output them as a single continuous video file.

[1178] Device Role

[1179] Image projection

[1180] The device receives the final video file sent from the server and displays it on the head-mounted display. The video is displayed in 360 degrees using the display device connected to the device. The video display format is automatically adjusted based on information from the emotion engine.

[1181] User Roles

[1182] Select a scene

[1183] Users can operate the interface to select specific scenes in the video. They can also check their own emotional state through the interface. The emotion engine analyzes the user's facial expressions and voice and provides the optimal video display format.

[1184] Viewing

[1185] The user wears a head-mounted display and watches a 360-degree visualization. The emotion engine recognizes emotions in real time and can highlight important information and issue warnings if the user is feeling stressed.

[1186] Specific examples

[1187] For example, if an operator is monitoring machinery in a factory and the emotion engine detects that the operator's stress level is high, the HMD image will instantly change based on that information. A 360-degree visualization of the machine that detected the abnormality will be displayed, highlighting the important parts. Below is an example of a prompt using a generative AI model:

[1188] "What is the best way to acquire real-time video data from machines in this factory and display it appropriately based on the operator's emotions?"

[1189] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1190] Step 1:

[1191] Acquiring video data

[1192] The server retrieves and reads the specified video data file. A video file name such as movie.mp4 is given as input data. The server reads this file using a video processing library such as OpenCV or FFmpeg and obtains the video frames. The output is the image data for each frame.

[1193] Step 2:

[1194] Grid division of video data

[1195] The server divides the captured video frame into a grid pattern. The input data is the video frame captured in the previous step. The video frame is divided into 10x10 meshes, and each mesh is saved as separate data. The output is the data for each mesh area.

[1196] Step 3:

[1197] Interpolation between meshes

[1198] The server uses a generative AI method to fill in the gaps between each divided mesh region. The input data is the data for the divided mesh regions. Using a generative AI model (e.g., GPT-3.5), it analyzes the boundaries of each mesh and generates pixels in the gaps based on the surrounding pixel information. The output is the mesh region data after the completion of the filling process.

[1199] Step 4:

[1200] Emotion recognition

[1201] The server uses an emotion engine to recognize the user's emotions in real time. The input data is the user's facial expression and voice data. The emotion engine (e.g., Azure Cognitive Services) analyzes the data and recognizes the user's emotional state. The output is the recognized emotional state data.

[1202] Step 5:

[1203] Select a scene

[1204] The user operates the interface to select a specific scene. The input data is a list of scenes. For each scene selected by the user, the server identifies the video data for that scene. The output is the video data for the selected scene.

[1205] Step 6:

[1206] Video display format conversion

[1207] The server converts the video into a 360-degree full-screen or central rectangular display depending on the selected scene. The input data is the video data of the selected scene and the emotional state data recognized in the previous step. The server converts the video into the optimal display format based on the emotional state. The output is the video data in the converted display format.

[1208] Step 7:

[1209] Merging and saving the final footage

[1210] The server combines and saves the converted video frames as a single continuous file. The input data is the converted video frame data. Using a video processing library, it concatenates these frames and saves them in a format such as converted_movie.mp4. The output is the final video file.

[1211] Step 8:

[1212] Image projection

[1213] The terminal receives the final video file sent from the server and displays it on the head-mounted display. The input data is the saved final video file. This data is sent to the HMD, which displays a 360-degree visualized video. The output is the video displayed on the HMD.

[1214] Step 9:

[1215] User reviews

[1216] The user wears a head-mounted display and watches a dynamically adjusted 360-degree video. The input data is the video displayed on the HMD. The output is the user's emotional response, which is then fed back to the emotion engine.

[1217] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1218] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1219] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1220] [Fourth embodiment]

[1221] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1222] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1223] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1224] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1225] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1226] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1227] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1228] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1229] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1230] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1231] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1232] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1233] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1234] This invention is a system that uses generative AI to convert existing video data into images that can be projected onto a 360-degree spherical screen. This system is composed of a server, terminals, and users working together, and each step plays a different role.

[1235] Server Roles

[1236] Acquiring video data

[1237] The server retrieves the specified video data file. For example, if a video file named movie.mp4 is specified, the server reads it and starts processing.

[1238] Mesh division of video

[1239] The server divides the acquired video data into a grid. For example, it divides a video frame into a 10x10 mesh and saves each mesh in a format that can be processed individually.

[1240] Interpolation between meshes

[1241] The server uses generation AI to fill in the gaps between the divided meshes. The generation AI analyzes the boundaries of each mesh and generates a complementary image based on information from surrounding pixels. This results in a smooth transition between images, making it suitable for spherical projection.

[1242] Scene selection and processing

[1243] Through the interface, the user selects whether to display a particular scene (e.g., ocean or space) in full 360-degree screen or in a central rectangular view. This selection information is sent to the server, which then performs the appropriate processing for each scene.

[1244] Merging and saving the final footage

[1245] The server merges all the processed frames and generates the final spherical projection video file, which is saved on the server in the format converted_movie.mp4, for example.

[1246] Device Role

[1247] Image projection

[1248] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. Using a projector connected to the terminal, the video is expanded and displayed on a 360-degree spherical screen, allowing the audience to experience the video in all directions.

[1249] User Roles

[1250] Select a scene

[1251] The user selects a specific scene from the video through an interface (e.g., dedicated software or a web application). Based on this selection, the system determines whether that scene should be displayed as a full 360-degree screen, while the other scenes should be displayed as a rectangular area in the center.

[1252] Viewing

[1253] Users can view images projected onto a spherical screen in the planetarium. The immersive feeling provided by 360-degree images allows users to enjoy a new visual experience that is different from that of a conventional flat screen.

[1254] Specific examples

[1255] For example, if a user wants to view an "ocean" scene in full 360-degree screen, the user selects the "ocean" scene through the interface. Based on this information, the server uses AI to generate the "ocean" scene for full 360-degree screen viewing, while other scenes (e.g., indoor scenes) are displayed as rectangular images in the center. Finally, the generated spherical projection video file is sent from the server to the device and projected onto the spherical screen of the planetarium. This process allows the user to enjoy a new visual experience.

[1256] Through these processing steps, the system of the present invention can easily convert existing images for spherical projection, providing a highly immersive experience.

[1257] The processing flow will be explained below.

[1258] Step 1: Acquire video data

[1259] The server retrieves the specified video data file. For example, if the file name is movie.mp4, the server reads this file and starts processing it. The video file is split into frames using a video processing library, and each frame is saved in a buffer.

[1260] Step 2: Meshing the image

[1261] The server divides each loaded frame into a grid. Specifically, it divides the frame into 10x10 meshes and treats each mesh as individual data. This process is performed using an image processing library.

[1262] Step 3: Interpolation between meshes

[1263] The server uses generative AI to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates pixels to fill in the gaps based on information from surrounding pixels. This filling process smooths the connections between the meshes, generating a continuous image suitable for spherical projection.

[1264] Step 4: Select a Scene

[1265] The user operates an interface to select specific scenes in the video, for example, using dedicated software or a web application. For each scene, the user can choose between a full-screen 360-degree view or a central rectangular view.

[1266] Step 5: Decide on the scene display format

[1267] The server receives the scene information selected by the user and determines the display format for each scene based on that information: certain scenes are processed for a full 360-degree screen display, while others are processed for a central rectangular display.

[1268] Step 6: Merging and saving the final footage

[1269] The server then combines all the processed frames to generate the final spherical projection video file. The resulting video file is saved on the server in the format converted_movie.mp4, for example. To do this, a video processing library is used to connect the frames and output them as a single continuous video file.

[1270] Step 7: Projecting the image

[1271] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. Using a projector connected to the terminal, the video is enlarged and displayed on a 360-degree spherical screen. This projection allows the audience to enjoy the video in all directions.

[1272] Step 8: Watch the video

[1273] Users can view images projected onto a spherical screen in the planetarium. The 360-degree images provide a new visual experience that differs from that of conventional flat screens.

[1274] Example 1

[1275] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1276] Conventional video projection technology has difficulty adapting existing flat images to a 360-degree spherical screen, making it difficult to maintain the continuity and smoothness of the image. Furthermore, there are limited ways for users to select specific scenes and determine the optimal display method. This makes it difficult to provide a highly immersive visual experience.

[1277] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1278] In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, a generating AI means for filling in the gaps between each grid-shaped division, means for the user to select a specific scene, means for converting the video into a 360-degree full screen or a central rectangular display depending on the selected scene, means for integrating and saving the final video for spherical projection, and means for projecting the integrated video onto a spherical screen. This makes it possible to smoothly convert existing video for spherical projection and provide the optimal display method based on the user's selection.

[1279] The "means for acquiring the specified video data" refers to the means used by the server to read and process the video file specified by the user.

[1280] The "means for dividing image data into a grid" is a means for dividing acquired image data based on a certain mesh size, making it easier to process the data individually.

[1281] The "generative AI means for filling in the gaps between each grid-like divided area" is a means for using a generative AI model to fill in the gaps that occur between divided meshes, thereby maintaining the continuity and smoothness of the image.

[1282] The "means for the user to select a particular scene" refers to an interface or operation means for the user to select a particular scene or location that he or she wishes to display.

[1283] The "means for converting the video into a 360-degree full screen or a rectangular display in the center according to the selected scene" is a means for converting the display format of the video into a 360-degree full screen or a rectangular display in the center based on the user's selection.

[1284] The "means for integrating and storing final spherical projection images" refers to a means for integrating interpolated and transformed image frames and storing them in a format suitable for a spherical screen.

[1285] The "means for projecting the integrated image onto a spherical screen" refers to a means for projecting the integrated and stored image onto a spherical screen using a projection device.

[1286] This invention is a system that uses a generative AI model to convert existing video data into images that can be projected onto a 360-degree spherical screen. This system is composed of a server, a terminal, and a user working together, and each step plays a different role.

[1287] Server Roles

[1288] Acquiring video data

[1289] The server retrieves the video data file specified by the user. For example, if the user specifies a file called "movie.mp4" through the interface, the server reads this file from the file system and saves it in a temporary directory.

[1290] Mesh division of video

[1291] The server divides the acquired video data into 10x10 meshes for each frame. This process uses an image processing library (e.g., OpenCV). The divided mesh parts are saved in a format that can be processed individually.

[1292] Interpolation between meshes

[1293] The server uses a generative AI model (e.g., GAN or VQ-VAE) to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates a complementary image based on information from surrounding pixels. This results in a smooth transition between the images, making them suitable for spherical projection.

[1294] Scene selection and processing

[1295] The user uses an interface (dedicated software or web application) to select the specific scene they want to view. For example, if the user wants to view an "ocean" scene in full 360 degrees, the server receives this information and generates a video centered on that scene.

[1296] Merging and saving the final footage

[1297] The server merges all processed frames and generates the final spherical projection video file, which is saved in MP4 format and named "converted_movie.mp4" for example.

[1298] Device Role

[1299] Image projection

[1300] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. A projection device (e.g., a 4K projector) connected to the terminal projects the video onto the 360-degree spherical screen, allowing the audience to experience the video in all directions.

[1301] User Roles

[1302] Select a scene

[1303] Through the interface, the user selects a particular scene in the video, and based on this selection, the server performs the appropriate processing for that scene and generates the video.

[1304] Viewing

[1305] Users can view images projected onto a spherical screen in the planetarium. The immersive feeling provided by 360-degree images allows users to enjoy a new visual experience that is different from that of a conventional flat screen.

[1306] Examples and prompts

[1307] For example, if a user wants to use an interface to view a full-screen 360-degree "ocean" scene, they can use the following prompt:

[1308] "Create a full-screen 360-degree ocean scene."

[1309] Based on this prompt, the server uses AI to generate the "ocean" scene, and other scenes (such as indoor scenes) are displayed as a central rectangular image. The generated spherical projection image file is sent from the server to the terminal, which then projects the image onto the spherical screen of the planetarium.

[1310] Through these processing steps, the system of the present invention can easily convert existing images for spherical projection, providing a highly immersive experience.

[1311] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1312] Program processing flow

[1313] Step 1: Acquire video data

[1314] Specific behavior:

[1315] The server reads the video file (e.g., "movie.mp4") specified by the user through the interface. It checks whether the file exists and whether read permissions are correct. The video data is saved in a temporary directory.

[1316] Input and Output:

[1317] Input: User-specified video file name ("movie.mp4")

[1318] Data processing and data calculation: Checking file existence and reading

[1319] Output: Video data is saved to a temporary directory

[1320] Step 2: Mesh division of video data

[1321] Specific behavior:

[1322] The server divides the acquired video data into 10x10 meshes for each frame. This process uses an image processing library (e.g., OpenCV). The divided mesh parts are saved in a format that can be processed individually.

[1323] Input and Output:

[1324] Input: Video data saved in the temporary directory

[1325] Data processing and data calculation: Divide the video frame into 10x10 meshes

[1326] Output: The divided mesh parts are saved as temporary files.

[1327] Step 3: Interpolation between meshes

[1328] Specific behavior:

[1329] The server uses a generative AI model (e.g., GAN or VQ-VAE) to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates a fill-in image based on the surrounding pixel information.

[1330] Input and Output:

[1331] Input: Divided mesh parts

[1332] Data processing and data calculation: Fill gaps between meshes with generated AI

[1333] Output: The completed mesh part is resaved

[1334] Step 4: Select and process the scene

[1335] Specific behavior:

[1336] The user selects the specific scene they want to display using an interface (dedicated software or web application). The server receives this information and generates a video centered on that scene. For example, if you want to display an "ocean" scene in full 360 degrees, the generation AI will perform the interpolation process.

[1337] Input and Output:

[1338] Input: User's scene selection information (e.g., "ocean" scene)

[1339] Data processing and data calculation: Generate images based on selected scenes and process them with AI

[1340] Output: Video data of the processed scene

[1341] Step 5: Merging and saving the final footage

[1342] Specific behavior:

[1343] The server will then combine all the interpolated and processed frames to generate the final spherical projection video file, which will be saved in MP4 format and named "converted_movie.mp4" for example.

[1344] Input and Output:

[1345] Input: Interpolated and processed frames

[1346] Data processing and data calculation: Frame integration and video file generation

[1347] Output: Final spherical projection video file

[1348] Step 6: Projecting the image

[1349] Specific behavior:

[1350] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium using a projection device (e.g., a 4K projector) connected to the terminal.

[1351] Input and Output:

[1352] Input: Final spherical projection image file

[1353] Data processing and data calculation: Playback of video files

[1354] Output: Image projected onto a spherical screen

[1355] Step 7: Select a Scene

[1356] Specific behavior:

[1357] The user selects a particular scene through the interface, and when the user selects a scene and clicks the "View" button on the interface, this information is sent to the server, which processes the scene.

[1358] Input and Output:

[1359] Input: User scene selection information

[1360] Data Processing and Data Computation: Scene-Based Video Processing

[1361] Output: Footage of the processed scene

[1362] Step 8: Viewing

[1363] Specific behavior:

[1364] Users can enjoy the 360-degree immersive experience by viewing images projected onto a spherical screen in a planetarium.

[1365] Input and Output:

[1366] Input: Image projected onto a spherical screen

[1367] Data processing and calculation: None

[1368] Output: The immersive visual experience the user experiences

[1369] In this way, the system performs appropriate processing at each step to adapt existing footage to a 360-degree spherical screen, providing users with a new visual experience.

[1370] (Application example 1)

[1371] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1372] Conventional video streaming services require special equipment and complex operations to enhance the user's sense of immersion, often limiting the quality of the visual experience. Furthermore, there is a need for an efficient method to convert existing video data into 360-degree spherical video and enable real-time viewing.

[1373] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1374] In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, a generation AI means for filling in gaps between each grid-shaped division, means for the user to select a specific scene, means for converting the video into a 360-degree full screen or rectangular display depending on the selected scene, means for integrating and saving the final video for spherical projection, means for streaming the final video for spherical projection to the user's terminal, and means for performing video data interpolation processing based on a 10x10 mesh. This allows users to convert existing video data into high-quality 360-degree spherical video and enjoy an immersive visual experience in real time without the need for special equipment or complex operations.

[1375] The "means for acquiring the designated video data" is a means by which the server receives the video data file that the user wants to view and starts processing.

[1376] The "means for dividing image data into a grid" is a means for dividing the acquired image data into a grid of a certain size and saving each mesh in a processable form.

[1377] The "generative AI means for filling in the gaps between each grid-like divided area" is a means for using a generative AI to analyze the gaps between divided meshes and generate a filled image based on surrounding pixel information.

[1378] The "means for the user to select a particular scene" is a means for the user to select a particular scene that he or she wishes to display through an interface.

[1379] "Means for converting the video into a 360-degree full screen or rectangular display according to the selected scene" refers to means for converting the video into a 360-degree full screen display or a central rectangular display based on the selected scene.

[1380] The "means for integrating and saving the final spherical projection image" is a means for integrating all the processed frames and saving them on a server as a final spherical projection image file.

[1381] The "means for projecting an image onto a spherical screen" refers to a means for projecting the final image received by the terminal onto a spherical screen, thereby displaying an image that spreads in all directions.

[1382] The "means for streaming the final spherical projection image to the user's terminal" refers to a means for transmitting the final image generated by the server to the user's terminal in real time, enabling continuous playback.

[1383] "Means for performing video data interpolation processing based on a 10x10 mesh" refers to a means for dividing a video frame into a 10x10 mesh and using a generation AI to interpolate the gaps between the meshes.

[1384] This invention is a system that uses generative AI to convert existing video data into video that can be projected onto a 360-degree spherical screen, enabling real-time viewing on a smartphone. The system is composed of a server, terminals, and users working together.

[1385] Server Roles

[1386] Acquiring video data

[1387] The server retrieves the specified video data file. For example, when a user specifies the file name of a movie or video they want to watch, the server reads the video data and begins processing.

[1388] Mesh division of video

[1389] The server divides the acquired video data into a 10x10 grid. Each mesh is saved in a format that can be processed individually. This allows subsequent processing by the generation AI to be performed efficiently.

[1390] Interpolation between meshes

[1391] The server uses a generative AI model to fill in the gaps between the divided meshes. The generative AI analyzes the boundaries of each mesh and generates a complementary image based on surrounding pixel information. This results in a smooth transition between images, making them suitable for spherical projection.

[1392] Scene selection and processing

[1393] The user selects a specific scene (e.g., ocean or space) through the interface. Based on this selection, the server displays the specific scene as a full 360-degree screen and processes the other scenes as a rectangular display in the center.

[1394] Merging and saving the final footage

[1395] The server then combines all the processed frames to generate the final spherical projection video file, which is then stored on the server and subsequently streamed to the user's device in real time.

[1396] Device Role

[1397] Projecting and viewing images

[1398] The device receives the final video file sent from the server and displays it as a 360-degree spherical image, allowing users to view images in all directions using their smartphones. This provides a high-quality immersive experience without the need for special equipment.

[1399] User Roles

[1400] Select a scene

[1401] Users select the specific scenes they want to watch through dedicated software or a web application, and the server performs the necessary video processing based on this selection.

[1402] Watching videos

[1403] Users can watch 360-degree spherical images streamed in real time on their smartphones, providing a new visual experience that is different from that of conventional flat screens.

[1404] Hardware and software used

[1405] Hardware: Servers, smartphones

[1406] Software: Python, OpenCV, MoviePy, FastAI

[1407] Data processing and calculation

[1408] Data Acquisition and Meshing

[1409] Using Python and OpenCV, we take the video data and divide it into a 10x10 mesh.

[1410] Interpolation between meshes

[1411] Using FastAI, the generative AI model complements the mesh boundaries of the video frame to generate smooth images.

[1412] Video integration and storage

[1413] The final video is integrated and saved using OpenCV and MoviePy, and data for streaming distribution is generated.

[1414] Examples and prompts

[1415] As a concrete example, let's say a user wants to display an "ocean" scene in full 360-degree screen. The user selects the "ocean" scene using the interface. Based on this information, the server uses AI to generate the "ocean" scene for full 360-degree screen display, and processes other scenes (e.g., indoor scenes) as rectangular displays in the center.

[1416] Prompt Sentence Examples

[1417] "Divide the following video data into a 10x10 mesh, and use a generative AI model to fill in the gaps between the meshes to generate an image for 360-degree spherical projection."

[1418] This system allows users to easily enjoy high-quality 360-degree spherical images.

[1419] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1420] Step 1:

[1421] Acquiring video data

[1422] The server retrieves the video data file (e.g., movie.mp4) specified by the user. Specifically, the user specifies the video file they want to watch, and the server reads that file. In this process, the path and name of the video file are input data, and the retrieved video is output.

[1423] Step 2:

[1424] Mesh division of video

[1425] The server divides the acquired video data into a grid. Specifically, it uses Python's OpenCV library to divide the video frame into a 10x10 mesh and saves each mesh in a processable format. The input of this step is the acquired video frame, and the output is each divided mesh.

[1426] Step 3:

[1427] Interpolation between meshes

[1428] The server uses a generative AI model to fill in the gaps between the meshes. Specifically, it uses the FastAI library to have the generative AI model analyze the boundaries of each mesh and generate a filled image based on the surrounding pixel information. The input to this step is the segmented mesh data, and the output is the filled image data.

[1429] Step 4:

[1430] Select a scene

[1431] The user selects a specific scene through the interface. Specifically, the user selects the scene they want to display (e.g., "ocean" or "space") using dedicated software or a web application, and this information is sent to the server. The input of this step is the user's selection information, and the output is the selected scene information.

[1432] Step 5:

[1433] Scene-specific video processing

[1434] The server processes the video according to the selected scene. Specifically, it converts certain scenes into a 360-degree full-screen display based on the selection information, and other scenes into a rectangular display with the selection information at the center. The video conversion is performed using Python and OpenCV. The input of this step is the selected scene information and the interpolated video data, and the output is the converted video data.

[1435] Step 6:

[1436] Merging and saving the final footage

[1437] The server merges all processed video frames to generate the final spherical projection video file. Specifically, it uses OpenCV and MoviePy to merge the frames into a single video file. The input of this step is the converted video data, and the output is the merged final video file.

[1438] Step 7:

[1439] Video streaming

[1440] The server then streams the final video file to the user's device. Specifically, it uses HTTP streaming technology to send the video to the user's smartphone in real time. The input of this step is the final video file, and the output is the video played on the user's device.

[1441] Step 8:

[1442] Projecting and viewing images

[1443] The user's device displays the final video received from the server as a 360-degree spherical video. Specifically, a smartphone application plays the video, allowing the user to view the omnidirectional video without the need for special equipment. The input of this step is the streamed video data, and the output is the user's visual experience.

[1444] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1445] This invention is a system that uses generative AI to convert existing video data into video that can be projected onto a 360-degree spherical screen, and combines it with an emotion engine to realize a video display that responds to the user's emotions. This system is composed of a server, terminals, and users working together, and each step plays a different role.

[1446] Server Roles

[1447] Acquiring video data

[1448] The server retrieves and reads the specified video data file. For example, if the file name is movie.mp4, the server reads this file and starts processing.

[1449] Mesh division of video

[1450] The server divides the acquired video data into a grid. Specifically, it divides the video frame into 10x10 meshes and saves each mesh as separate data.

[1451] Interpolation between meshes

[1452] The server uses generative AI to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates pixels to fill in the gaps based on information from surrounding pixels. This filling process smooths the connections between the meshes, generating a continuous image suitable for spherical projection.

[1453] Emotion engine integration

[1454] The server is integrated with an emotion engine that recognizes the user's emotions. This emotion engine analyzes changes in the user's facial expressions and voice to recognize emotions in real time, and automatically adjusts the video scene selection and display format based on that information.

[1455] Scene selection and processing

[1456] In addition to the user selecting a specific scene, the emotion engine monitors the user's emotions in real time, and based on this information, the video will either switch to a full 360-degree screen view or adjust to a central rectangular view depending on the emotion.

[1457] Merging and saving the final footage

[1458] The server then combines all the processed frames to generate the final spherical projection video file. The resulting video file is saved on the server in the format converted_movie.mp4, for example. To do this, a video processing library is used to connect the frames and output them as a single continuous video file.

[1459] Device Role

[1460] Image projection

[1461] The device receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. The video is enlarged and displayed on a 360-degree spherical screen using a projector connected to the device. The video display format is automatically adjusted based on information from the emotion engine.

[1462] User Roles

[1463] Select a scene

[1464] Users can operate the interface to select specific scenes in the video. They can also check their own emotional state through the interface. The emotion engine analyzes the user's facial expressions and voice and provides the optimal video display format.

[1465] Viewing

[1466] Users view images projected onto a spherical screen in the planetarium. The emotion engine recognizes emotions in real time, and if the user is feeling emotional, it switches to a full-screen 360-degree display of a specific scene. In this way, users can enjoy a highly immersive visual experience.

[1467] Specific examples

[1468] For example, if a user is viewing a 360-degree full-screen "ocean" scene in a planetarium, the emotion engine will analyze the user's facial expressions and, if it detects signs of emotion, automatically switch to a full-screen 360-degree display of the next scene. If the emotion engine's analysis determines that the user is bored, it will switch to a rectangular display in the center and provide other content. This process ensures that the user always receives the optimal visual experience.

[1469] Through these processing steps, the system of the present invention can easily convert existing images for spherical projection, providing a highly immersive experience. Furthermore, by integrating an emotion engine, it is possible to realize image display that adapts to the user's emotions.

[1470] The processing flow will be explained below.

[1471] Step 1: Acquire video data

[1472] The server retrieves the specified video data file. Specifically, if the file name is movie.mp4, the server reads this file and uses a video processing library (e.g., OpenCV) to store each frame in a buffer.

[1473] Step 2: Meshing the image

[1474] The server divides each loaded frame into a grid. Specifically, it divides each frame into 10x10 meshes and saves each mesh as separate data. This separates the fine details of the image, making it easier to perform subsequent interpolation processing.

[1475] Step 3: Interpolation between meshes

[1476] The server uses generative AI to fill in the gaps between the divided meshes. The generative AI model analyzes the boundaries of each mesh and generates pixels to fill in the gaps based on information from surrounding pixels. This step smooths the connections between the meshes, resulting in a continuous image suitable for spherical projection.

[1477] Step 4: Obtaining emotion data

[1478] The emotion engine integrated into the server acquires the user's facial expression and voice data to recognize the user's emotions in real time. This involves capturing the user's facial expressions and voice using a camera and microphone, and analyzing the emotions using emotion recognition algorithms.

[1479] Step 5: Select a Scene

[1480] The user operates the interface to select a specific scene. When the user selects a scene, the emotion engine's analysis results are also taken into account. For example, if the user is emotional, the emotion engine will automatically suggest an enhanced scene display.

[1481] Step 6: Decide on the scene display format

[1482] The server determines the display format for each scene based on the scene information selected by the user and the analysis results of the emotion engine. Specifically, if the server recognizes that the user is moved, it displays a specific scene in full 360-degree screen, spreading the image in all directions instead of displaying it in a rectangular format in the center.

[1483] Step 7: Merging and saving the final footage

[1484] The server then combines all the processed frames to generate the final spherical projection video file (e.g., converted_movie.mp4). The resulting video file is then concatenated using a video processing library to create a single continuous video file, which is then saved on the server.

[1485] Step 8: Projecting the image

[1486] The device receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. Using a projector connected to the device, the video is expanded and displayed on a 360-degree spherical screen. The video is optimally adjusted based on the analysis of the emotion engine.

[1487] Step 9: Watch the video

[1488] Users watch images projected onto a spherical screen in the planetarium. The emotion engine analyzes the user's emotions in real time, adjusting the projection of specific scenes if the user is excited, for example, to optimize the user experience. In this way, users can enjoy a highly immersive visual experience.

[1489] Example 2

[1490] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1491] Conventional video display systems have had difficulty optimizing video display in response to user emotions. Furthermore, complex data processing and advanced technology are required to project existing video data in a form suitable for a spherical screen. This can result in a degradation of the quality of the user experience. The purpose of this invention is to solve these problems and realize a highly immersive video display that responds to the user's emotions.

[1492] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, and a generation AI means for filling in gaps between each grid-shaped division. This makes it possible to process existing video data into an image suitable for spherical projection. Furthermore, by including means for a user to select a specific scene, means for converting the image into a 360-degree full screen or rectangular display depending on the selected scene, means for integrating and saving the final image for spherical projection, and means for projecting the image onto the spherical screen, it is possible to realize an image display that is optimal for the scene and emotion specified by the user. Furthermore, by including emotion engine means for recognizing the user's emotion and means for automatically adjusting the image display format according to the user's emotion using the emotion engine means, it is possible to realize real-time adjustment of the image display according to the user's emotion.

[1493] Definitions of important words

[1494] "Specified video data" refers to a video or video file specified by a user or the system prior to entering the system.

[1495] "Generative AI means" refers to a method or device that uses artificial intelligence technology to fill in gaps or missing parts of video data and generate smooth, continuous video.

[1496] "User selection means" refers to a method or device that allows a user to select a particular scene or video portion from the interface.

[1497] "Emotion engine means" refers to technology or a device that analyzes a user's facial expressions, voice, and other biometric information to recognize the user's emotional state in real time.

[1498] "Means for transforming video into a 360-degree full-screen or rectangular display" refers to technology or devices that adapt video data to an all-around display format or a specific shape based on the user's visual preferences.

[1499] "Means for integrating and saving the final spherical projection image" refers to the technology and equipment for integrating and saving the results of each process into a single continuous image file.

[1500] "Means for projecting an image onto a spherical screen" refers to a method or device for displaying the generated spherical projection image on a screen capable of displaying 360 degrees.

[1501] "Means for automatically adjusting the video display format according to the user's emotions using emotion engine means" refers to technology or devices that dynamically change the video display format based on the results of user emotion recognition, providing an optimal visual experience.

[1502] MODE FOR CARRYING OUT THE INVENTION

[1503] This invention is a system that uses a generative AI model to convert existing video data into video that can be projected onto a 360-degree spherical screen, and combines it with an emotion engine to realize video display that responds to the user's emotions. This system is composed of a server, terminals, and users working together, and the roles of each are explained in detail below.

[1504] Server Roles

[1505] Acquiring video data

[1506] The server retrieves and reads the specified video data file. Specifically, if the user specifies a file called "movie.mp4", the server reads the file from a specified directory and loads it into memory as byte data. This retrieval process uses standard file I / O operations and byte stream processing libraries (e.g., Python's os module and io module).

[1507] Mesh division of video

[1508] The server divides the acquired video data into a grid for each frame. Specifically, it uses an image processing library (e.g., OpenCV library) to divide each frame into a 10x10 mesh. Each mesh portion is temporarily saved as individual small image data. This allows the next interpolation process to proceed smoothly.

[1509] Interpolation between meshes

[1510] The server uses a generative AI model to fill in the gaps between each divided mesh. To do this, the server uses a generative AI model (e.g., GANs or AutoEncoder) to analyze surrounding pixel information and generate pixels to fill in the gaps. Specifically, the server trains the model using a deep learning framework such as PyTorch or TensorFlow, and outputs the filled-in parts based on the boundary pixel information of each mesh. This filling process smooths the connections between the meshes.

[1511] Emotion engine integration

[1512] The server is integrated with an emotion engine that analyzes facial and voice data sent from the user's webcam and microphone. This analysis uses face recognition APIs and voice analysis libraries (e.g., OpenCV, Dlib, and Google's Cloud Speech-to-Text API). The analyzed emotion information is used to adjust the video display format.

[1513] Scene selection and processing

[1514] The server selects an appropriate video scene based on the user's interface operations and data from the emotion engine. For example, if the user selects an "ocean scene" on the interface, the server obtains frame data for that scene, and if the emotion engine detects the user's emotion, it changes the video to 360-degree full-screen mode.

[1515] Merging and saving the final footage

[1516] The server reintegrates all meshes after the interpolation process is completed to generate the final spherical projection video file. It then uses a video processing library (e.g., FFmpeg) to concatenate each frame and save it on the server as a single continuous video file (e.g., converted_movie.mp4). This process ensures that all frames are displayed continuously.

[1517] Device Role

[1518] Image projection

[1519] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium. The image is enlarged using a 360-degree projector (e.g., a fisheye projector) connected to the terminal. The display format of the image is automatically adjusted according to the user's emotions, providing the user with an enhanced visual experience.

[1520] User Roles

[1521] Select a scene

[1522] The user operates the interface to select a specific scene. The user can also check their own emotional state through the interface. The emotion engine analyzes the user's facial expressions and voice and provides a video display format that best suits the user's emotions.

[1523] Viewing

[1524] Users view images projected onto a spherical screen in a planetarium. The emotion engine monitors the user's emotions in real time, and if the user is emotionally moved, the scene switches to a full 360-degree screen. This process allows users to view the images with a high level of immersion.

[1525] Examples and prompts

[1526] For example, consider a case where a user is viewing an "ocean scene" in a planetarium while viewing it in full 360-degree screen. If the emotion engine analyzes the user's facial expression and recognizes signs of emotion, it automatically switches to displaying the next scene in full 360-degree screen. Conversely, if it determines that the user is bored, it switches the image to a rectangular display in the center and provides other content.

[1527] An example prompt is, "Convert the movie file movie.mp4 into a 360-degree video and use the emotion engine to adjust the display format according to the user's emotion."

[1528] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1529] Program processing flow

[1530] Step 1: Acquire video data

[1531] The server retrieves the specified video data file (input: specified file path). Based on the prompt, it retrieves the specified file (e.g., movie.mp4) from the directory and loads it into memory as byte data. Specifically, it opens the file using Python's os and io modules and reads its contents as a data stream. The output is the video data loaded into memory.

[1532] Step 2: Meshing the image

[1533] The server divides the acquired video data into a grid for each frame (input: video data in memory). Specifically, it uses an image processing library (e.g., OpenCV) to divide each frame into a 10x10 mesh. Each mesh portion is temporarily saved as individual image data. Specifically, it divides the frame into an array, and then divides it into a grid using a cv2 function. The output is a list of each divided mesh image.

[1534] Step 3: Interpolation between meshes

[1535] The server uses a generative AI model to fill in the gaps between each divided mesh (input: list of divided mesh images). Using a generative AI model (e.g., GANs or AutoEncoder), it analyzes surrounding pixel information and generates pixels in the gaps. Specifically, it uses PyTorch or TensorFlow to input the boundary data of each mesh into the AI ​​model, which then outputs the filled pixel information. The output is a continuous mesh image with the gaps filled in.

[1536] Step 4: Integrating the Emotion Engine

[1537] The server integrates an emotion engine that analyzes the user's emotions in real time (input: user's facial expression data and voice data). The emotion engine uses face recognition APIs and voice analysis libraries (e.g., OpenCV and Dlib, Google Cloud Speech-to-Text API) to analyze the user's emotional state. The emotion data is used to adjust the video display. Specifically, it analyzes real-time streams from the webcam and microphone and outputs the type and intensity of emotions as data. The output is the analyzed emotion information.

[1538] Step 5: Select and process the scene

[1539] The server selects and processes video scenes based on user operations and data from the emotion engine (input: user-selected data and emotion information). Specifically, it obtains the scene selected in the user interface (e.g., "ocean scene") and switches to a 360-degree full-screen or rectangular display depending on the information from the emotion engine. Specifically, it analyzes the emotion data and dynamically changes the video display format. The output is video data adjusted to the optimal display format.

[1540] Step 6: Merging and saving the final footage

[1541] The server reintegrates all meshes after completion of the interpolation process and generates the final spherical projection video file (input: list of interpolated mesh images). It uses a video processing library (e.g., FFmpeg) to concatenate each frame and save it as a single continuous video file on the server. Specifically, it concatenates and encodes the frames and saves the result in an appropriate file format (e.g., converted_movie.mp4). The output is the saved spherical projection video file.

[1542] Step 7: Projecting the image

[1543] The terminal receives the final video file sent from the server and projects it onto the spherical screen of the planetarium (input: final video file). A 360-degree compatible projector (e.g., a Fisheye projector) is used to enlarge and display the image. Specifically, the projector's control software is used to decode the video file and project it onto the screen. The output is a 360-degree image displayed on the spherical screen.

[1544] Step 8: Select and watch a scene

[1545] The user operates the interface to select a specific scene and view it in the planetarium (input: user selection). The emotion engine monitors the user's emotions in real time and switches between full-screen and rectangular display depending on the emotion. Specifically, the scene display is adjusted based on the user's operations and emotional data. The output is the visual experience the user views.

[1546] (Application example 2)

[1547] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1548] Conventional systems that project video data onto a spherical screen have the problem of being unable to display images adaptively according to the user's emotions. Furthermore, in industrial environments, there has been a lack of systems that dynamically highlight important information when operators monitor the status of machinery. This has led to problems such as increased operator stress and reduced work efficiency.

[1549] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1550] In this invention, the server includes means for acquiring specified video data, means for dividing the video data into a grid, a generation AI means for filling in gaps between each grid-shaped division, means for the user to select a specific scene, means for converting the video into a 360-degree full screen or rectangular display depending on the selected scene, means for integrating and saving the final video for spherical projection, means for projecting the video onto the spherical screen, means for recognizing the user's emotions using an emotion engine and adaptively displaying information accordingly, and means for displaying the video on a head-mounted display in an industrial environment and dynamically emphasizing important information. This enables adaptive video display according to the user's emotions, improving the efficiency of operator monitoring work in industrial environments and reducing stress.

[1551] The "means for acquiring designated video data" is a method for acquiring and reading a video data file designated in advance by the user or the system.

[1552] The "means for dividing video data into a grid" is a method for dividing acquired video data into a fixed grid pattern and storing the data as individual grids.

[1553] "Generative AI means" refers to artificial intelligence technology that analyzes the gaps between divided grid-like areas and fills them in based on surrounding pixel information.

[1554] The "means for the user to select a particular scene" is a method in which the user operates the interface to select a particular video scene that he or she wishes to view.

[1555] "Means for converting the video to a 360-degree full screen or rectangular display depending on the selected scene" refers to a method for converting the video so that the scene selected by the user is displayed in a 360-degree full screen or a rectangular display in the center.

[1556] The "means for integrating and saving the final spherical projection image" is a method for combining the generated image frames into a single continuous file and saving it in a format that can be projected onto a spherical screen.

[1557] "Means for projecting images onto a spherical screen" refers to a method using a projector or hardware for projecting a stored image file onto a spherical screen.

[1558] An "emotion engine" is a technology that analyzes a user's facial expressions and voice and recognizes their emotional state in real time.

[1559] "Means for adaptively displaying information" refers to a method for dynamically adjusting the information and images displayed based on the user's emotional state.

[1560] "Industrial environment" refers to an environment where industrial activities take place, such as a factory or manufacturing site.

[1561] A "head-mounted display" is a device worn on the user's head that displays images and information directly into the user's field of vision.

[1562] The "means for dynamically highlighting important information" is a method for detecting important information in an industrial environment in real time and highlighting it to the user.

[1563] This invention is a system that uses a head-mounted display (HMD) in an industrial environment to visualize images in a factory in 360 degrees and dynamically highlight information according to the operator's emotions. An embodiment of this system will be described in detail below.

[1564] Server Roles

[1565] The server performs the following main functions:

[1566] Acquiring video data

[1567] The server acquires and reads the specified video data file, using a video processing library such as OpenCV or FFmpeg.

[1568] Mesh division of video

[1569] The server divides the acquired video data into a grid. Specifically, it divides the video frame into 10x10 meshes and saves each mesh as separate data.

[1570] Interpolation between meshes

[1571] The server uses generative AI to fill in the gaps between the divided meshes. Using a generative AI model (such as GPT-3.5), it analyzes the boundaries of each mesh and generates pixels to fill in the gaps based on information from surrounding pixels. This filling process smooths the connections between the meshes, generating a continuous image suitable for 360-degree projection.

[1572] Emotion engine integration

[1573] The server is integrated with an emotion engine that recognizes the user's emotions. For example, using Azure Cognitive Services and TensorFlow, it analyzes changes in the user's facial expressions and voice to recognize emotions in real time. Based on this information, the video scene selection and display format are automatically adjusted.

[1574] Scene selection and processing

[1575] In addition to the user selecting a specific scene, the emotion engine monitors the user's emotions in real time, and based on this information, the video will either switch to a full 360-degree screen view or adjust to a central rectangular view depending on the emotion.

[1576] Merging and saving the final footage

[1577] The server then combines all the processed frames to generate the final 360-degree projection video file. The resulting video file is saved on the server in a format such as converted_movie.mp4. This is done by using a video processing library to connect the frames and output them as a single continuous video file.

[1578] Device Role

[1579] Image projection

[1580] The device receives the final video file sent from the server and displays it on the head-mounted display. The video is displayed in 360 degrees using the display device connected to the device. The video display format is automatically adjusted based on information from the emotion engine.

[1581] User Roles

[1582] Select a scene

[1583] Users can operate the interface to select specific scenes in the video. They can also check their own emotional state through the interface. The emotion engine analyzes the user's facial expressions and voice and provides the optimal video display format.

[1584] Viewing

[1585] The user wears a head-mounted display and watches a 360-degree visualization. The emotion engine recognizes emotions in real time and can highlight important information and issue warnings if the user is feeling stressed.

[1586] Specific examples

[1587] For example, if an operator is monitoring machinery in a factory and the emotion engine detects that the operator's stress level is high, the HMD image will instantly change based on that information. A 360-degree visualization of the machine that detected the abnormality will be displayed, highlighting the important parts. Below is an example of a prompt using a generative AI model:

[1588] "What is the best way to acquire real-time video data from machines in this factory and display it appropriately based on the operator's emotions?"

[1589] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1590] Step 1:

[1591] Acquiring video data

[1592] The server retrieves and reads the specified video data file. A video file name such as movie.mp4 is given as input data. The server reads this file using a video processing library such as OpenCV or FFmpeg and obtains the video frames. The output is the image data for each frame.

[1593] Step 2:

[1594] Grid division of video data

[1595] The server divides the captured video frame into a grid pattern. The input data is the video frame captured in the previous step. The video frame is divided into 10x10 meshes, and each mesh is saved as separate data. The output is the data for each mesh area.

[1596] Step 3:

[1597] Interpolation between meshes

[1598] The server uses a generative AI method to fill in the gaps between each divided mesh region. The input data is the data for the divided mesh regions. Using a generative AI model (e.g., GPT-3.5), it analyzes the boundaries of each mesh and generates pixels in the gaps based on the surrounding pixel information. The output is the mesh region data after the completion of the filling process.

[1599] Step 4:

[1600] Emotion recognition

[1601] The server uses an emotion engine to recognize the user's emotions in real time. The input data is the user's facial expression and voice data. The emotion engine (e.g., Azure Cognitive Services) analyzes the data and recognizes the user's emotional state. The output is the recognized emotional state data.

[1602] Step 5:

[1603] Select a scene

[1604] The user operates the interface to select a specific scene. The input data is a list of scenes. For each scene selected by the user, the server identifies the video data for that scene. The output is the video data for the selected scene.

[1605] Step 6:

[1606] Video display format conversion

[1607] The server converts the video into a 360-degree full-screen or central rectangular display depending on the selected scene. The input data is the video data of the selected scene and the emotional state data recognized in the previous step. The server converts the video into the optimal display format based on the emotional state. The output is the video data in the converted display format.

[1608] Step 7:

[1609] Merging and saving the final footage

[1610] The server combines and saves the converted video frames as a single continuous file. The input data is the converted video frame data. Using a video processing library, it concatenates these frames and saves them in a format such as converted_movie.mp4. The output is the final video file.

[1611] Step 8:

[1612] Image projection

[1613] The terminal receives the final video file sent from the server and displays it on the head-mounted display. The input data is the saved final video file. This data is sent to the HMD, which displays a 360-degree visualized video. The output is the video displayed on the HMD.

[1614] Step 9:

[1615] User reviews

[1616] The user wears a head-mounted display and watches a dynamically adjusted 360-degree video. The input data is the video displayed on the HMD. The output is the user's emotional response, which is then fed back to the emotion engine.

[1617] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1618] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1619] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1620] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1621] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1622] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1623] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1624] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1625] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1626] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1627] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1628] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1629] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1630] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1631] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1632] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1633] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1634] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1635] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1636] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1637] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1638] The following is further disclosed regarding the above embodiment.

[1639] (Claim 1)

[1640] means for acquiring designated video data;

[1641] means for dividing video data into a grid;

[1642] A generating AI means for filling in gaps between each grid-like divided area;

[1643] a means for a user to select a particular scene;

[1644] means for converting the video to a 360-degree full screen or rectangular display depending on the scene selected;

[1645] a means for integrating and storing the final spherical projection image;

[1646] A system including means for projecting an image onto a spherical screen.

[1647] (Claim 2)

[1648] The system described in claim 1, characterized in that it uses a generation AI means that complements gaps between meshes to analyze the boundaries of each grid-shaped divided area and generate a complemented image based on surrounding pixel information.

[1649] (Claim 3)

[1650] 2. The system of claim 1, wherein certain scenes are displayed as a full 360-degree screen, and other scenes are converted to be displayed as a rectangular centered view.

[1651] "Example 1"

[1652] (Claim 1)

[1653] means for acquiring designated video data;

[1654] means for dividing video data into a grid;

[1655] A generating AI means for filling in gaps between each grid-like divided area;

[1656] a means for a user to select a particular scene;

[1657] A means to convert the video to a full 360-degree screen or a central rectangular display depending on the scene selected;

[1658] a means for integrating and storing the final spherical projection image;

[1659] The system includes a means for projecting the integrated image onto a spherical screen.

[1660] (Claim 2)

[1661] The system described in claim 1, characterized in that it uses a generation AI means that complements gaps between meshes to analyze the boundaries of each grid-shaped divided area and generate a complemented image based on surrounding pixel information.

[1662] (Claim 3)

[1663] 2. The system of claim 1, wherein certain scenes are displayed as a full 360-degree screen, and other scenes are converted to be displayed as a rectangular centered view.

[1664] "Application Example 1"

[1665] (Claim 1)

[1666] means for acquiring designated video data;

[1667] means for dividing video data into a grid;

[1668] A generating AI means for filling in gaps between each grid-like divided area;

[1669] a means for a user to select a particular scene;

[1670] means for converting the video to a 360-degree full screen or rectangular display depending on the scene selected;

[1671] a means for integrating and storing the final spherical projection image;

[1672] means for projecting an image onto a spherical screen;

[1673] means for streaming the final spherical projection image to a user terminal;

[1674] A means for performing interpolation processing of video data based on a 10x10 mesh;

[1675] A system including:

[1676] (Claim 2)

[1677] The system described in claim 1 is characterized in that it uses a generation AI means that fills in gaps between meshes, analyzes the boundaries of each grid-shaped divided area, generates a complemented image based on surrounding pixel information, and finally streams it to the user's terminal.

[1678] (Claim 3)

[1679] The system described in claim 1, characterized in that certain scenes are displayed as 360-degree full screen, and other scenes are converted to be displayed as rectangular images in the center, and the converted images are streamed to the user's terminal.

[1680] "Example 2: Combining Emotion Engines"

[1681] Claims

[1682] (Claim 1)

[1683] means for acquiring designated video data;

[1684] means for dividing video data into a grid;

[1685] A generating AI means for filling in gaps between each grid-like divided area;

[1686] a means for a user to select a particular scene;

[1687] means for converting the video to a 360-degree full screen or rectangular display depending on the scene selected;

[1688] a means for integrating and storing the final spherical projection image;

[1689] means for projecting an image onto a spherical screen;

[1690] emotion engine means for recognizing the emotion of a user;

[1691] means for automatically adjusting a video display format according to a user's emotion by using an emotion engine means;

[1692] A system including:

[1693] (Claim 2)

[1694] The system according to claim 1, characterized in that it uses a generation AI means to analyze the boundaries of each grid-shaped divided area and generate a complementary image based on surrounding pixel information.

[1695] (Claim 3)

[1696] 2. The system according to claim 1, wherein the emotion engine means analyzes the user's emotions in real time and automatically adjusts whether a particular scene is displayed as a 360-degree full screen or as a rectangular display in the center.

[1697] "Application example 2 when combining emotion engines"

[1698] (Claim 1)

[1699] means for acquiring designated video data;

[1700] means for dividing video data into a grid;

[1701] A generating AI means for filling in gaps between each grid-like divided area;

[1702] a means for a user to select a particular scene;

[1703] means for converting the video to a 360-degree full screen or rectangular display depending on the scene selected;

[1704] a means for integrating and storing the final spherical projection image;

[1705] means for projecting an image onto a spherical screen;

[1706] means for recognizing a user's emotion using an emotion engine and adaptively displaying information in accordance with the emotion;

[1707] A system that includes a means for displaying images on a head-mounted display in an industrial environment and dynamically highlighting important information.

[1708] (Claim 2)

[1709] The system described in claim 1, characterized in that it uses a generation AI means that complements gaps between meshes to analyze the boundaries of each grid-shaped divided area and generate a complemented image based on surrounding pixel information.

[1710] (Claim 3)

[1711] 2. The system of claim 1, wherein certain scenes are displayed as a full 360-degree screen, and other scenes are converted to be displayed as a rectangular centered view. [Explanation of symbols]

[1712] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for acquiring designated video data; means for dividing video data into a grid; A generating AI means for filling in gaps between each grid-like divided area; a means for a user to select a particular scene; means for converting the video to a 360-degree full screen or rectangular display depending on the scene selected; a means for integrating and storing the final spherical projection image; A system including means for projecting an image onto a spherical screen.

2. The system described in claim 1, characterized in that it uses a generation AI means that complements gaps between meshes to analyze the boundaries of each grid-shaped divided area and generate a complemented image based on surrounding pixel information.

3. 2. The system according to claim 1, wherein a specific scene is displayed as a 360-degree full screen, and other scenes are converted to be displayed as a rectangular image in the center.

Images