Information processing method, program, and information processing system

Abstract

To make it possible to generate a more natural moving image from an image.SOLUTION: An information processing method includes: an acquisition step of obtaining an image; a segmenting step of performing segmentation for each element composing a subject included in the image; a prompt generation step of generating a prompt for each element segmented in the segment step; and a moving image generation step of generating a moving image based on the prompt for each element which is generated in the prompt generation step.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing method, a program, and an information processing system. [Background technology]

[0002] Cited Document 1 discloses a video generation program that uses AI to generate videos from text and still images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6783479 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the case of the above-mentioned prior art, although videos corresponding to categories are generated, only broad categories such as people and animals are available, and it is difficult to generate desired videos with finer details, such as only the eyes or only the mouth of a person. Therefore, in order to create more natural videos, it is desirable to generate videos with finer details individually.

[0005] In consideration of the above, an object of the present invention is to provide an information processing method, a program, and an information processing system that can generate more natural moving images. [Means for solving the problem]

[0006] According to one embodiment of an information processing method, the information processing method is executed by an information processing device and includes an acquisition step of acquiring an image, a segmentation step of segmenting the image into elements that make up a subject, a prompt generation step of generating a prompt for each element segmented in the segmentation step, and a video generation step of generating a video based on the prompt for each element generated in the prompt generation step.

[0007] According to one embodiment of the program, an information processing device is caused to execute an information processing method including an acquisition step for acquiring an image, a segmentation step for segmenting each element constituting a subject contained in the image, a prompt generation step for generating a prompt for each element segmented in the segmentation step, and a video generation step for generating a video based on the prompt for each element generated in the prompt generation step.

[0008] According to one embodiment, the information processing system is executed by an information processing device, and includes an acquisition step for acquiring an image, a segmentation step for segmenting the image into elements that constitute a subject, a prompt generation step for generating a prompt for each element segmented in the segmentation step, and a video generation step for generating a video based on the prompt for each element generated in the prompt generation step. [Effects of the Invention]

[0009] According to one embodiment, more natural moving images can be generated. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server according to an embodiment. [Figure 3]FIG. 2 is a diagram illustrating an example of a hardware configuration of a user terminal according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of a server according to an embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of a functional configuration of a user terminal according to an embodiment. [Figure 6] 10 is a flowchart illustrating an example of processing executed by the information processing system according to the embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of an image before a moving image is generated according to the embodiment. [Figure 8] FIG. 10 illustrates an example of segmentation according to an embodiment. [Figure 9] FIG. 10 illustrates an example of segmentation according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] An embodiment of an information processing system according to the present invention will be described below with reference to Figures 1 to 5. In each drawing, the same or equivalent components and parts are denoted by the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0012] (System Overview) First, an overview of an information processing system 10 according to this embodiment will be described. The information processing system 10 according to this embodiment is a system that can generate moving images from images. In this embodiment, the information processing system 10 is a system that can generate prompts for each element of a subject in an image according to the situation in the image, and generate moving images based on the generated prompts.

[0013] (System Configuration) Fig. 1 is a diagram showing an example of the configuration of an information processing system 10 according to this embodiment. As shown in Fig. 1, the information processing system 10 according to this embodiment includes a user terminal 2 and a server 1, which are communicably connected to each other via a network N. The network N is, for example, a wired local area network (LAN), a wireless LAN, the Internet, a public line network, a mobile data communication network, or a combination thereof.

[0014] The user terminal 2 is an example of an information processing device that performs operations for inputting and displaying various information. The user terminal 2 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, a wearable device, or a combination of these. In this embodiment, an information processing device equipped with a photographing function is used as an example.

[0015] The server 1 is an example of an information processing device that generates prompts based on the situation of an image input by the user terminal 2 and generates a video from the image. The server 1 may be a PC (Personal Computer), a smartphone, a tablet terminal, a server device, a microcomputer, or a combination of these. The specific configuration and operation of the server 1 will be described later.

[0016] (Hardware configuration - Server) 2 is a block diagram showing the hardware configuration of the server 1. The server 1 includes a processor 101, a memory 102, a storage 103, and a communication I / F 104, which are communicably connected to each other via a bus B.

[0017] The processor 101 controls each component of the server 1 and realizes the functions of the server 1 by loading various programs stored in the storage 103 into the memory 102 and executing them. The programs executed by the processor 101 include, but are not limited to, an OS (Operating System) and various programs described below. Execution of these programs by the processor 101 realizes part of the state visualization method according to this embodiment. The processor 101 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), or a combination thereof.

[0018] The memory 102 is, for example, a read-only memory (ROM), a random access memory (RAM), or a combination thereof. The ROM is, for example, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a combination thereof. The RAM is, for example, a dynamic random access memory (DRAM), a static random access memory (SRAM), a magnetoresistive random access memory (MRAM), or a combination thereof.

[0019] The storage 103 stores an OS, various programs (described later), and various data. The storage 103 is, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), a storage class memory (SCM), or a combination of these.

[0020] The communication I / F 104 is an interface for connecting the server 1 to external devices including the user terminal 2 via the network N and controlling communication. The communication I / F 104 is, for example, an adapter compliant with Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), Ethernet (registered trademark), or optical communication (e.g., Fibre Channel), but is not limited to these.

[0021] (Hardware configuration - user terminal) 3 is a block diagram showing the hardware configuration of the user terminal 2. The user terminal 2 includes a processor 201, a memory 202, a storage 203, a communication I / F 204, an input / output I / F 205, an input device 206, and an output device 207, which are communicatively connected to each other via a bus B. The communication I / F 204 is an interface for connecting the user terminal 2 to external devices including the server 1 via a network N and for controlling communication. The communication I / F 204 is, for example, an adapter compliant with Bluetooth (registered trademark), Wi-Fi (registered trademark), ZigBee (registered trademark), Ethernet (registered trademark), or optical communication (e.g., Fibre Channel), but is not limited to these.

[0022] The input / output I / F 205 is an interface for connecting an input device 206 and an output device 207 to the user terminal 2. The input device 206 is, for example, a mouse, a keyboard, a touch panel, a microphone, a scanner, a camera, various sensors, operation buttons, or a combination of these. The output device 207 as a user interface is, for example, a display, a projector, a printer, a speaker, a vibrator, or a combination of these. In this embodiment, as an example, the output device 207 and the input device 206 are an integrally configured touch panel display.

[0023] In this embodiment, the program may be written to the memory 202 or the storage 203 during the manufacturing stage of the server 1, or may be provided to the server 1 via the network N. Alternatively, the program may be provided to the server 1 via a non-transitory computer-readable recording medium such as a disk medium (not shown).

[0024] (Functional configuration - Server) Next, the functional configuration of the server 1 will be described. Fig. 4 is a diagram showing an example of the functional configuration of the server 1. When executing various programs, the server 1 realizes various functions using the above hardware resources. The server 1 has a communication unit 11, a storage unit 12, and a control unit 13 as the functional configuration realized by the server 1. Each functional configuration is realized by the processor 101 reading and executing a program 121 stored in the memory 102 or storage 103.

[0025] The control unit 13 includes an acquisition unit 131 that acquires an image sent from the user terminal 2 and a segmentation unit 132 that segments the subject of the image acquired by the acquisition unit 131 into elements. The control unit 13 also includes a prompt generation unit 133 that generates prompts for the elements segmented by the segmentation unit 132, and a video generation unit 134 that generates a video in accordance with the prompt generated by the prompt generation unit 133. The acquisition unit 131 acquires a situation related to an image input by the user terminal 2 and registers the situation information in the situation DB 122. Here, the situation information may be, for example, weather information (rain, wind, sunny, cloudy, etc.), time of day (morning, noon, evening, night, etc.), memories of when the image was captured, a specific space such as underwater, etc., or a combination thereof. The prompt generation unit 133 generates a prompt in accordance with the situation information registered in the situation DB 122. The video generation unit 134 generates a video from the prompt generated by the prompt generation unit 133 using a video generation model. These will be described in detail later.

[0026] (Functional configuration - User terminal) Next, the functional configuration of the user terminal 2 will be described. Figure 5 is a diagram showing an example of the functional configuration of the user terminal 2. When executing various programs, the user terminal 2 realizes various functions using the above-mentioned hardware resources. The user terminal 2 has, as the functional configuration realized by the user terminal 2, a communication unit 21, a storage unit 22 in which a program 221 is stored, and a control unit 23. Each functional configuration is realized by the processor 201 reading and executing the program 221 stored in the memory 202 or the storage 203. The control unit 23 includes an information acquisition control unit 231 that acquires information sent from the server 1, and a display unit 232 that displays the acquired information on the output device 207.

[0027] Next, the flow of processing of the information processing system according to this embodiment will be described with reference to Fig. 6 to Fig. 9. Fig. 6 is a flowchart of processing executed by the information processing system 10 according to this embodiment. Fig. 7 is a diagram showing an example of an image before generating a moving image according to this embodiment, and Fig. 8 and Fig. 9 are each diagrams showing an example of segmentation according to this embodiment.

[0028] First, in step S101, the acquisition unit 131 acquires an image sent from the user terminal 2. At this time, the acquisition unit 131 registers the situation information input at the user terminal 2 in the situation DB 122. Next, in step S102, the segmentation unit 132 segments the subject of the image acquired in step S101. Here, the segmentation in step S102 will be described with reference to Figs. 7 to 9.

[0029] FIG. 7 is a diagram illustrating an example of an image acquired in step S101. In this embodiment, people T1 to T3, a bird T4, a tree T5, and a sun T6 are displayed as subjects. FIG. 8 is a diagram illustrating an example of segmenting and displaying each of the subjects in the image shown in FIG. 7. As shown in FIG. 8, the segmentation unit 132 segments each of the people T1 to T3, the bird T4, the tree T5, and the sun T6 in FIG. 7 by surrounding them with frames. Furthermore, in this embodiment, the subjects are segmented in more detail. FIG. 9 is a diagram illustrating detailed segmentation of person T1 among the subjects. Specifically, the segmentation unit 132 divides the image into elements of the head F1, right arm F2, left arm F3, torso F4, right leg F5, left leg F6, and cloak F7 of person T1.

[0030] In this embodiment, person T1 is described as an example. However, the other subjects, namely people T2 and T3, bird T4, tree T5, and sun T6, are also divided and segmented into their respective elements in the same manner. For example, bird T4 is divided into its head, legs, and feathers, and tree T5 is divided into its trunk, branches, and leaves. Segmentation is not limited to these, and more detailed classification may be used. Furthermore, segmentation may be performed by, for example, segmenting each element based on image features, or by preparing a trained model and using that model to segment each element. Alternatively, segmentation may be performed using a general-purpose model that can be used without prior training. In this embodiment, segmentation is performed for each subject, and then segmentation is performed for each element of the subject. However, this is not limited to this. For example, segmentation for each subject may be omitted, and only segmentation for each element may be performed.

[0031] Returning to the flowchart of FIG. 6, in step S103, it is determined whether all of the subjects have been segmented into elements. If all of the subjects in the image have been segmented into elements (YES), the process proceeds to step S104. Next, in step S104, the acquisition unit 131 acquires situation information registered in the situation DB 122. Next, in step S105, the prompt generation unit 133 generates a prompt for each element segmented in step S102 based on the situation information acquired in step S104. Specifically, using the person T1 in FIG. 9 as an example, a case will be described in which the situation information is daytime (daytime), the sun is out, and the wind is strong. In this case, the prompt generation unit 133 generates a prompt for the head F1 such that hair flutters in the wind and a shadow is cast according to the height of the sun. Furthermore, the prompt generation unit 133 generates a prompt for the right arm F2, left arm F3, torso F4, right leg F5, and left leg F6 such that clothes flutter in the wind and a shadow is cast according to the height of the sun. Furthermore, for the cape F7, the prompt generation unit 133 generates a prompt that makes the cape flutter wildly in the wind. Regarding the generation of the prompt, for example, prompts for each element according to the situation may be registered in a database in advance, and the prompt may be generated from the database. Alternatively, the prompt may be generated from the text of each element and situation information using LLM (Large Language Models).

[0032] The generation of prompts is not limited to this embodiment. For example, in the case of the sea, a prompt is generated to move the wave height and the like in accordance with the situation. Also, for example, if it is evening, a prompt is generated to move the sun in the direction of setting sun to create a sunset sky. Furthermore, for example, if it is a happy memory, a prompt is generated to make the person's facial expression change to a smile. In this way, a prompt is generated that matches the situation included in the situation information. This situation information is information input by the user and can be any setting, whether rough or detailed.

[0033] Next, in step S106, the moving image generation unit 134 generates moving images of each element based on the prompt generated in step S105. Specifically, the moving image generation unit 134 generates moving images of each element based on the prompts for each element, namely, the head F1, right arm F2, left arm F3, torso F4, right leg F5, left leg F6, and cape F7 of the person T1 in FIG. 9, generated in step S105. Next, in step S107, the moving image generation unit 134 unifies the moving images generated for each element to generate a moving image. At this time, the moving image generation unit 134 generates a moving image of the image acquired in step S101 by taking into account the overall balance of each element based on situation information.

[0034] In this embodiment, the video generation unit 134 generates the entire video in a unified manner in step S107, but if the video generation unit 134 generates multiple videos based on a prompt in step S106, a video more desired by the user can be generated. In other words, multiple videos are generated for each element in step S106, and the user can select from the multiple videos generated for each element via the user terminal 2, thereby generating a video more desired by the user.

[0035] In this manner, in this embodiment, when generating a moving image from an acquired image, a prompt can be generated for each element of the subject of the image according to the situation in the image, and the moving image can be generated.

[0036] As described above, according to this embodiment, a more natural moving image can be generated by segmenting an image into individual elements of a subject and generating a moving image for each element. While this embodiment describes a method of generating a moving image directly from an image, the present invention is not limited to this method. For example, a moving image may be generated by converting the image into an animation-like image. In this case, for example, a process of converting the subject into an animation-like image may be added after the image is acquired in step S101. Furthermore, the situation information may include not only general situations, but also situations that are not directly related to photographs, such as wizards, pirates, underwater scenes, and space.

[0037] Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc. within the scope of achieving the object of the present invention are included in the present invention.

[0038] Furthermore, for example, the above-described series of processes can be executed by hardware or software. In other words, the functional configuration is merely an example and is not particularly limited. That is, it is sufficient that the information processing system has the function of being able to execute the above-described series of processes as a whole, and there is no particular limit to the type of functional block used to realize this function. Furthermore, the location of the functional block is also not particularly limited and may be arbitrary. For example, a functional block of a server may be transferred to a user terminal, etc. Conversely, a functional block of a user terminal may be transferred to a server, etc. Furthermore, one functional block may be configured as a single piece of hardware, a single piece of software, or a combination thereof.

[0039] Furthermore, for example, when a series of processes is executed by software, the programs constituting the software are installed onto a computer or the like from a network or a recording medium. The computer may be a computer incorporated into dedicated hardware. Furthermore, the computer may be a computer capable of executing various functions by installing various programs thereon, such as a server, a general-purpose smartphone, or a personal computer.

[0040] Furthermore, for example, the recording medium containing such a program may be configured not only as a removable medium (not shown) that is distributed separately from the device main body in order to provide the program to the user, but also as a recording medium that is provided to the user in a state that is pre-installed in the device main body.

[0041] In this specification, the steps of describing a program to be recorded on a recording medium include not only processes that are performed chronologically in accordance with the order, but also processes that are not necessarily performed chronologically but are performed in parallel or individually. In addition, in this specification, the term "system" refers to an overall device that is made up of a plurality of devices, a plurality of means, etc. [Explanation of symbols]

[0042] 10 Information Processing Systems 1 server 2. User terminal 131 Acquisition Department 132 segment section 133 Prompt Generation Unit 134 Video Generation Unit

Claims

1. An information processing method executed by an information processing device, an acquisition step of acquiring an image; a segmentation step of segmenting the image into elements constituting the subject; a prompt generation step of generating a prompt based on situation information for each of the elements segmented in the segment step; a video generation step of generating a video for each of the elements using a video generation model based on the prompt for each of the elements generated in the prompt generation step, and unifying the video for each of the elements based on the situation information while taking into account the overall balance of each element to generate a video of the image; Including, In the video generation step, when a plurality of videos are generated for each of the elements, a video selected from the plurality of videos is unified to generate a video of the image. Information processing methods.

2. In the prompt generation step, a prompt is generated from the text of each element and situation information using a large-scale language model. The information processing method according to claim 1 .

3. In the information processing device, an acquisition step of acquiring an image; a segmentation step of segmenting the image into elements constituting the subject; a prompt generation step of generating a prompt based on situation information for each of the elements segmented in the segment step; a video generation step of generating a video for each of the elements using a video generation model based on the prompt for each of the elements generated in the prompt generation step, and unifying the video for each of the elements in consideration of the overall balance of each element based on the situation information to generate a video of the image, In the video generation step, when generating a video for each element, if multiple videos are generated, a program for executing an information processing method in which a video selected from the multiple videos is unified to generate a video of the image.

4. An information processing system executed by an information processing device, an acquisition step of acquiring an image; a segmentation step of segmenting the image into elements constituting the subject; a prompt generation step of generating a prompt based on situation information for each of the elements segmented in the segment step; a video generation step of generating a video for each of the elements using a video generation model based on the prompt for each of the elements generated in the prompt generation step, and unifying the video for each of the elements based on the situation information while taking into account the overall balance of each element to generate a video of the image; In the video generation step, when a plurality of videos are generated for each of the elements, a video selected from the plurality of videos is unified to generate a video of the image. Information processing system.

Images