Information processing system, information processing method, and program

Abstract

To allow a user to create a replaced video image that the user privately enjoys, through a simpler operation.SOLUTION: In a CPU 11 of a management server 1 which constitutes an information processing system S, an extraction unit 32 extracts video images of one or more scenes from a base video image, and an association unit 34 associates types of the scenes for each of the extracted video images of the scenes. A replacement content designation receiving unit 54 receives a designation of content for replacement processing. A processing target designation receiving unit 55 receives a designation of a target to be subjected to replacement processing, out of the extracted video images of the scenes. A replacement unit 35 performs designated replacement processing on the designated video image of the scene. An output control unit 36 outputs the replaced video image of the scene, as a replaced video image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] BACKGROUND ART Techniques for replacing a part of image data with another image are known (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-217232 Summary of the Invention [Problem to be solved by the invention]

[0004] While conventional techniques allow users to create their preferred replacement videos by replacing a portion of a base video with another video, there is a need for a system that allows users to create replacement videos that they can enjoy privately with simpler operations.

[0005] The present invention has been made in view of the above circumstances, and has an object to enable a user to create a replaced moving image that the user can enjoy privately with simpler operations. [Means for solving the problem]

[0006] In order to achieve the above object, one aspect of the present invention is to an extracting means for extracting one or more scenes from a base video; a replacement target object designation receiving means for receiving designation of a target object to be subjected to replacement processing from among the extracted moving images of the scene; a replacement object designation receiving means for receiving a designation of a replacement object to be replaced from the target object by the replacement process; a replacement means for performing the specified replacement process on the moving image of the scene; an output control means for outputting the moving image of the scene on which the replacement process has been performed as a replaced moving image; It is an information processing system having the above.

[0007] An information processing method and a program corresponding to the information processing system according to one aspect of the present invention are also provided as an information processing method and a program corresponding to the information processing system according to one aspect of the present invention. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a mechanism that enables a user to create a replaced video that the user can enjoy privately with simpler operations. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of an information processing system according to an embodiment of the present invention. [Figure 2] 2 is a block diagram showing an example of a hardware configuration of a management server that constitutes the information processing system of FIG. 1. FIG. [Figure 3] FIG. 2 is a functional block diagram showing an example of a functional configuration of a management server and a user terminal. [Figure 4] 1A is a flowchart showing an example of the flow of the management server's processing up to outputting the replaced video to the user terminal, and FIG. 1B is a flowchart showing an example of the flow of the management server's processing performed while the replaced video is being output (played) on the user terminal. [Figure 5] FIG. 10 is a diagram illustrating a specific example of this service. [Figure 6] FIG. 10 is a diagram illustrating a specific example of replacement processing. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, this embodiment will be described with reference to the drawings. <Information Processing System S> FIG. 1 is a diagram showing an example of the overall configuration of an information processing system S according to one embodiment of the present invention. The information processing system S is an information processing system configured to include a management server 1, user terminals 2-1 to 2-n (n is an integer value equal to or greater than 1), and a video manufacturer server 3, all connected via a network N. The network N is, for example, the Internet, a LAN (Local Area Network), or a VPN (Virtual Private Network). Hereinafter, when there is no need to explain each of the user terminals 2-1 to 2-n individually, they will be collectively referred to as "user terminals 2."

[0011] The information processing system S is an information processing system that makes it possible to provide a service (hereinafter referred to as "this service") that provides a user with a replacement video in which a part of a video designated by the user has been replaced. The video that is the subject of this service is assumed to be provided by a video manufacturer that produces and provides various types of video as a business. In other words, by using the video provided by the video manufacturer as the base video for the replacement process, only videos that meet certain compliance standards are subject to the replacement process.

[0012] However, the base video for the replacement process is not limited to video provided by a video manufacturer, but may be, for example, a video uploaded by a user. In this case, however, the information processing system S may set predetermined standards to scrutinize the video to be included in this service from the standpoint of compliance, etc. For example, illegally uploaded video may be excluded from this service. In this case, the information processing system S may use a predetermined AI model to determine whether the video provided by the video manufacturer is illegally uploaded. The predetermined AI model may be, for example, a large language model (LLM) that processes multimodal content including video. Note that the service provider that provides this service to users is assumed to have obtained prior permission from the video manufacturer and the user regarding the copyright of the video.

[0013] The information processing system S presents one or more videos provided by a video manufacturer to the user in a selectable manner. When a video to be subjected to replacement processing (hereinafter referred to as a "base video") is designated from the one or more videos presented to the user, the information processing system S extracts one or more scene videos from the designated base video. The information processing system S associates each of the extracted one or more scene videos with a scene type and presents them to the user in a selectable manner. Specific examples of scene types will be described later. The information processing system S may present a list of videos extracted as scenes to the user. Alternatively, the information processing system S may present scene types to the user, accept a selection of a scene type by the user, and present a list of extracted videos corresponding to the selected scene type.

[0014] The base video provided by the video manufacturer is generally used in common by multiple users. That is, the process for extracting scenes from the base video to be used in the information processing system S is performed before the user specifies the content of the replacement process.

[0015] When the content of the replacement process is specified by a user operation, the information processing system S performs replacement process based on the specified content of the replacement process. The content of the replacement process is determined by, for example, selecting a video to be replaced (a base video or a video of an extracted scene), specifying a target object to be replaced by the replacement process, specifying a replaced object to be replaced by the replacement process, etc. The information processing system S outputs the video of the scene on which the replacement process has been performed as a replaced video. The information processing system S controls the user terminal 2 that outputs the replaced video so that the replaced video cannot be saved.

[0016] When a predetermined operation is performed while a replaced video is being output (played) on the user terminal 2, the information processing system S can identify the replaced video being output (played) at the time the operation was performed and manage it as having low accuracy in replacement processing. Specific examples of the predetermined operation performed while a replaced video is being output (played) will be described later.

[0017] [Management Server 1] The management server 1 is an information processing device that serves as a server for managing the entire information processing system S. The management server 1 is capable of executing predetermined application programs that enable the information processing system S to be used. The management server 1 transmits various types of information to each of the user terminal 2, the video maker server 3, and the outside, and enables various processes to be executed. The management server 1 also acquires various types of information transmitted from each of the user terminal 2, the video maker server 3, and the outside, and enables various processes to be executed.

[0018] For example, the management server 1 identifies a base video image designated by a user from among one or more videos transmitted from the video manufacturer server 3, and extracts one or more video images of scenes from the base video image. Then, the management server 1 associates a scene type with each of the extracted one or more video images of scenes.

[0019] The management server 1 presents the extracted video of one or more scenes to the user in a selectable manner. When an operation to specify the content of the replacement process is performed on the user terminal 2, the management server 1 performs replacement process based on the specified content of the replacement process. The operation to specify the content of the replacement process includes, for example, an operation to upload a still image or video (hereinafter referred to as a "user image") capturing the face of the user that is the target of the replacement process and that will be included in the video after the replacement process, and an operation to specify the face of the user that is included in the user image and that will be the target of replacement.

[0020] As with base moving images, images uploaded as user images may be examined to determine whether they are eligible for the service, based on predetermined criteria established from the standpoint of compliance, etc. That is, for example, if a user uploads a still image or moving image of the user's own face, but the still image or moving image contains a face image of a celebrity, the information processing system S may not consider the image to be eligible for the service.

[0021] The operation of specifying the content of the replacement process includes an operation of specifying one or more video images of the presented one or more scenes to be subjected to the replacement process. The operation of specifying the content of the replacement process also includes an operation of specifying a character (e.g., the main character) to be subjected to the replacement process among the characters appearing in the base video, an operation of specifying which character's face should be replaced with the user's face, etc.

[0022] Here, the user's face to be replaced is extracted from the uploaded user image. The user image may be a photo in which the orientation, size, and brightness of the user's face as the subject are predetermined, such as an ID photo, or a photo captured in a single moment in a natural setting, such as a snapshot. However, if the user image is a snapshot, face recognition processing and cropping processing are performed. The user image may also be a moving image of the user. Note that if the user image contains multiple faces as subjects, the user can select which of the multiple facial images included in the user image to use as the replaced object.

[0023] When the content of the replacement process as described above is specified by a user operation, the management server 1 performs replacement process based on the specified content of the replacement process. Specifically, the management server 1 performs replacement process based on the specified content of the replacement process for each of the videos of one or more specified scenes. If there are multiple videos of scenes specified as targets for replacement process, the management server 1 may perform replacement process on the videos of the specified multiple scenes simultaneously in parallel, or in chronological order. Alternatively, the replacement process may be performed in random order depending on the amount of data, etc.

[0024] The management server 1 transmits the video of one or more scenes that have been subjected to the replacement process to the user terminal 2 as a replaced video, and causes the user terminal 2 to output the replaced video. At this time, the management server 1 causes the replaced video to be output in a manner that prevents it from being saved on the user terminal 2. Specifically, the management server 1 detects, among the user's operations on the user terminal 2, an operation to download the replaced video or an operation to take a screenshot of the replaced video, and performs control to prohibit these operations. Furthermore, the management server 1 makes it possible to output the replaced video on a management terminal (not shown) operated by a service provider, separate from the user terminal 2.

[0025] When a predetermined operation is performed while a replaced video is being output (played) on the user terminal 2, the management server 1 identifies the replaced video being output (played) at the time the operation was performed and manages it as having low replacement processing accuracy. Specifically, when a predetermined operation is performed while a replaced video is being output (played), the management server 1 identifies the replaced video being output (played) at the time the operation was performed, associates it with information indicating that the predetermined operation was performed, and stores the associated information. The management server 1 then feeds back information indicating the identified replaced video to a predetermined AI model. Here, in addition to identifying the replaced video, the management server 1 can also identify one or more frames constituting the replaced video being output (played) at the time the predetermined operation was performed, and associates it with information indicating that the predetermined operation was performed and stores the associated information.

[0026] The "predetermined operation" refers to, for example, an operation of pressing a so-called BAD button, which is pressed by a user viewing a replacement video when giving a low rating to the replacement video, or an operation of pressing a so-called SKIP button, which skips a predetermined number of seconds of output (playback) content. When a user presses the SKIP button, there is a possibility that the user has a negative impression of the video being viewed. For this reason, the information processing system S processes the user's operation of pressing the SKIP button in the same way as the operation of pressing the BAD button. Furthermore, compared to the operation of pressing the BAD button, the operation of pressing the SKIP button is less likely to disrupt the user's concentration while viewing the video, and therefore the user's judgment ability to judge the accuracy of the replacement process can be maintained at a high level.

[0027] By performing a predetermined operation while a replacement video is being output (played), the user can record that the replacement process for the replacement video being output (played) at the time the operation was performed is low in accuracy. Information indicating that the predetermined operation was performed while the replacement video was being output (played) is stored in association with the replacement video and fed back to a predetermined AI model. This feedback may involve learning to improve the accuracy of the replacement process only for the replacement video specified by the user. In other words, so-called overlearning of the AI ​​model may occur. "Overlearning" refers to a state in which accuracy cannot be achieved for unknown data, resulting from overfitting to training data. Note that in this service, the video of the extracted scene is fixed, so overlearning is acceptable, resulting in higher accuracy of the replacement process for only the video of that scene.

[0028] Here, the replacement process includes a preprocessing for analyzing the object to be replaced contained in the video of the scene (e.g., extracting feature points, etc.), and a process for replacing the object with a replaced object based on the results of the preprocessing. The management server 1 can execute (or cause to be executed) the preprocessing in advance before the user specifies the object to be replaced or the replaced object. The functional configuration and processing flow of the management server 1 will be described later.

[0029] [User terminal 2] The user terminal 2 is an information processing device operated by a user who uses this service. The user terminal 2 is configured, for example, as a smartphone, tablet terminal, personal computer, etc. The user terminal 2 is capable of executing a predetermined application program that enables use of the information processing system S. The user terminal 2 is capable of performing various processes based on various information transmitted from the management server 1 and the outside, and various information input by the user. The user terminal 2 is also capable of transmitting various information to both the management server 1 and the outside.

[0030] For example, the user terminal 2 displays information indicating one or more videos provided by a video manufacturer in a selectable manner. The user terminal 2 accepts an operation to designate one base video to be the target of replacement processing from one or more videos provided by the video manufacturer, and transmits the operation information to the management server 1. After that, the management server 1 extracts videos of scenes and associates the extracted scene videos with scene types, and the user terminal 2 displays information indicating the extracted one or more scene videos in a selectable manner.

[0031] The user terminal 2 accepts an operation to specify the content of the replacement process and transmits the operation information to the management server 1. The operation to specify the content of the replacement process includes an operation to specify by uploading a user image and an operation to specify the specific content of the replacement process. After that, the management server 1 performs replacement process based on the content of the replacement process specified by the user, and then outputs the replaced video to the user terminal 2. At this time, the replaced video output to the user terminal 2 is output in a format that cannot be saved in the user terminal 2, as described above.

[0032] The user terminal 2 outputs (plays) the replaced video. When a predetermined operation is performed while the replaced video is being output (played), the user terminal 2 accepts the operation and transmits the operation information to the management server 1. Here, the predetermined operation is, for example, pressing the BAD button or skip button described above. The functional configuration and processing flow of the user terminal 2 will be described later.

[0033] [Video Maker Server 3] The video maker server 3 is an information processing device that functions as a server capable of executing a predetermined application program that enables the use of the information processing system S. The video maker server 3 stores and manages videos produced by video makers. For example, the video maker server 3 enables the videos it manages to be transmitted to the management server 1.

[0034] The above-described processes performed by the management server 1, the user terminal 2, and the video maker server 3 constituting the information processing system S are merely examples. In other words, it is sufficient for the information processing system S as a whole to have the functions to realize the above-described processes, and some or all of the functions to realize the above-described processes may be shared or cooperated within the information processing system S.

[0035] For example, some or all of the functions of the management server 1 may be functions of other information processing devices in the information processing system S. Also, some or all of the functions of other information processing devices in the information processing system S may be functions of the management server 1. Furthermore, some or all of the functions of the management server 1 may be transferred to other servers (not shown). This promotes processing in the information processing system S as a whole and also makes it possible for processes to complement each other.

[0036] <Hardware configuration> [Hardware configuration of Management Server 1] FIG. 2 is a block diagram showing an example of the hardware configuration of the management server 1 that constitutes the information processing system S of FIG. The management server 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an output unit 16, an input unit 17, a memory unit 18, a communication unit 19, and a drive 20.

[0037] The CPU 11 executes various processes in accordance with programs recorded in the ROM 12 or programs loaded from the storage unit 18 into the RAM 13. The RAM 13 also stores data and the like required for the CPU 11 to execute various processes. The CPU 11, the ROM 12, and the RAM 13 are interconnected via a bus 14. An input / output interface 15 is also connected to this bus 14.

[0038] The input / output interface 15 is connected to an output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20. The output unit 16 is composed of a display, a speaker, etc., and outputs various types of information as images, sounds, etc. The input unit 17 is composed of a keyboard, a mouse, a touch panel, etc., and accepts input of various types of information. The storage unit 18 is composed of a hard disk, a DRAM (Dynamic Random Access Memory), etc., and stores various types of data. The communication unit 19 communicates with other devices via the above-mentioned network N, which is composed of the Internet, etc.

[0039] Removable media 21, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, is appropriately attached to the drive 20. Programs read from the removable media 21 by the drive 20 are installed in the storage unit 18 as needed. The removable media 21 can also store various data stored in the storage unit 18 in the same way as the storage unit 18.

[0040] [Hardware Configuration of User Terminal 2 and Video Maker Server 3] The user terminal 2 and the video maker server 3 have the same hardware configuration as the management server 1 shown in Fig. 2. That is, the user terminal 2 and the video maker server 3 each have a CPU, ROM, RAM, bus, input / output interface, output unit, input unit, memory unit, communication unit, drive, and removable media, which correspond to the CPU 11, ROM 12, RAM 13, bus 14, input / output interface 15, output unit 16, input unit 17, memory unit 18, communication unit 19, drive 20, and removable media 21, respectively, in Fig. 2.

[0041] <Functional configuration> [Functional configuration of management server 1] FIG. 3 is a functional block diagram showing an example of the functional configuration of the management server 1 and the user terminal 2. As shown in FIG. When the CPU 11 of the management server 1 operates, an acquisition unit 31, an information management unit 32 as a management means, an extraction unit 33 as an extraction means, a matching unit 34 as a matching means, a replacement unit 35 as a replacement means, and an output control unit 36 ​​as an output control means function.

[0042] Furthermore, various databases are provided in the storage unit 18 of the management server 1. For example, there are provided a provided video DB 41 that stores videos provided by the video maker server 3, a scene video DB 42 that stores videos of scenes extracted from base videos, a replaced video DB 43 that stores replaced videos that have been subjected to a replacement process, and a scene type DB 44 that stores scene types.

[0043] The acquisition unit 31 acquires various types of information. For example, the acquisition unit 31 acquires videos transmitted from the video manufacturer server 3. The acquisition unit 31 also acquires various types of operation information accepted and transmitted by the user terminal 2. Examples of the operation information transmitted from the user terminal 2 include operation information related to an operation to specify one base video to be replaced, operation information related to an operation to specify the content of the replacement, and information indicating that a predetermined operation (for example, an operation to press a BAD button or a SKIP button) was performed while the replaced video was being output (played).

[0044] The information management unit 32 stores and manages various types of information in the database of the storage unit 18. For example, the information management unit 32 stores and manages the video acquired by the acquisition unit 31 in a provided video DB 41. The information management unit 32 also stores and manages the video of scenes extracted by the extraction unit 33 (described later) in a scene video DB 42. The information management unit 32 also stores and manages the replaced video that has been subjected to a replacement process by the replacement unit 35 (described later) in a replaced video DB 43. The information management unit 32 also stores and manages a plurality of predetermined scene types in a scene type DB 44.

[0045] Furthermore, when a predetermined operation is performed while a replaced video is being output (played) from the user terminal 2, the information management unit 32 identifies and manages the replaced video being output (played) when the operation was performed. Specifically, the information management unit 32 associates the identified replaced video with information indicating that a predetermined operation was performed while the replaced video was being output (played), and stores and manages the information in the replaced video DB 43. The predetermined operation is, for example, pressing a BAD button or a SKIP button. The replaced video identified by the information management unit 32 is stored in association with information indicating that the predetermined operation was performed, and information indicating the identified replaced video is fed back to a predetermined AI model.

[0046] Furthermore, when a predetermined operation is performed while a replaced video is being output (played) from the user terminal 2, the information management unit 32 can identify and manage one or more frames that make up the replaced video being output (played) at the time the operation was performed. Note that a frame is a unit that indicates one still image when a video is captured as a series of multiple still images. By managing the frames that make up the replaced video, the information management unit 32 can identify positions where the accuracy of the replacement process should be improved, or can use the frames as a marker when the user wants to play the replaced video again.

[0047] The extraction unit 33 identifies a base video image designated by the user from among one or more videos provided by a video manufacturer, and extracts one or more video images of scenes from the base video image. The length (playback time) of each of the one or more video images of scenes extracted from the base video image is not particularly limited. For example, a video image of a scene with a length (playback time) of about 3 to 5 minutes may be extracted.

[0048] When extracting a moving image of a scene, the extraction unit 33 is capable of selecting and extracting a moving image of a scene that is easy to perform replacement processing. Specifically, the extraction unit 33 is capable of extracting a moving image of a scene based on the appearance (e.g., orientation, moving speed, etc.) of a part (e.g., a face of a character) included in a base moving image that can be a target of replacement processing. For example, the extraction unit 33 is capable of selecting and extracting a moving image of a scene that is easy to perform face recognition processing because the characters are facing forward, or a moving image of a scene that is easy to perform face recognition processing because the characters' movements are relatively gentle. Note that the process of selecting and extracting a moving image of a scene that is easy to perform replacement processing may be performed using a predetermined AI model.

[0049] The associating unit 34 associates a scene type with each of the one or more moving images of scenes extracted by the extracting unit 33. Predetermined scene types are stored in the scene type DB 44. Examples of scene types include everyday scenes, introduction scenes, development scenes, battle scenes, and conclusion scenes.

[0050] These scene types can also be managed as combinations. For example, assume that the information management unit 32 manages five scene types (daily life scene, introduction scene, development scene, battle scene, and conclusion scene) as one combination. In this case, the association unit 34 associates each of one or more scene videos extracted from the base video with one of the five scene types. Note that the process of associating each of one or more scene videos with a scene type may be performed using a predetermined AI model. In this case, before the association is performed, candidate associations may be suggested to the user. For example, videos of scenes corresponding to daily life scenes in which the characters move relatively calmly may be displayed in a selectable list, and then videos of scenes corresponding to battle scenes in which the characters move vigorously may be displayed in a selectable list.

[0051] The replacement unit 35 performs replacement processing based on the content of the replacement processing specified by the user. Specifically, the replacement unit 35 performs replacement processing based on the user image and the specific content of the replacement processing specified by the user. As the replacement processing, the replacement unit 35, for example, replaces the face of a character specified by the user among characters appearing in one or more videos of scenes specified by the user with the face of the user. In this case, the replacement unit 35 performs replacement processing by, for example, linking identification information associated with each face of one or more characters appearing in each of the videos of one or more scenes with identification information associated with the face of the user appearing in the user image. The identification information is, for example, a tag. The replacement unit 35 can also perform replacement processing on one or more base videos.

[0052] Specifically, for example, assume that five characters appearing in each of the moving images of one or more scenes are a protagonist, a heroine, and supporting characters A to C. In this case, the replacement unit 35 performs replacement processing by linking identification information associated with the faces of the five characters, for identifying whether the character is the protagonist, heroine, or supporting character A to C, with identification information associated with the face of the user appearing in the user image.

[0053] The facial recognition process used in the replacement process for replacing the faces of the characters and the process for associating identification information with the faces of the characters, which are performed by the replacement unit 35, may be performed using a predetermined AI model. Furthermore, the identification information itself may be suggested using the predetermined AI model. That is, the identification information itself indicating each of the main character, the heroine, and the supporting characters a to c may be suggested by the predetermined AI model.

[0054] Furthermore, the replacement unit 35 makes it possible to set a difference in the accuracy of the replacement process for a video image to be output (played) earlier and a video image to be output (played) later, among multiple types of scenes that are successively output (played) from the user terminal 2. For example, the replacement unit 35 makes it possible to set a higher accuracy of the replacement process for a video image to be output (played) earlier than for a video image to be output (played) later, among multiple types of scenes.

[0055] The output control unit 36 ​​controls the output of various types of information to the user terminal 2. For example, the output control unit 36 ​​controls the output of information indicating one or more videos provided by a video manufacturer to the user terminal 2. The output control unit 36 ​​also controls the output of information indicating one or more scenes of videos extracted by the extraction unit 33 to the user terminal 2.

[0056] The output control unit 36 ​​also controls the output of one or more replaced videos that have been subjected to the replacement process by the replacement unit 35 to the user terminal 2. At this time, the output control unit 36 ​​controls the output in a manner that prevents the user terminal 2 from saving the replaced videos.

[0057] [Functional configuration of user terminal 2] When the CPU of the user terminal 2 operates, an acquisition unit 51, a display control unit 52, a base image designation receiving unit 53, a replacement content designation receiving unit 54 as a content designation receiving means, a processing target designation receiving unit 55 as a processing target designation receiving means, and a transmission control unit 56 function.

[0058] The acquisition unit 51 acquires various types of information. For example, the acquisition unit 51 acquires information indicating one or more videos transmitted from the management server 1. The acquisition unit 51 also acquires information indicating one or more scenes transmitted from the management server 1.

[0059] The display control unit 52 controls the display of various types of information. Specifically, the display control unit 52 controls the display of various types of information on a display or the like of the output unit. For example, the display control unit 52 controls the display of information indicating one or more videos in a selectable manner on a display or the like. The display control unit 52 also controls the display of information indicating one or more scene videos in a selectable manner on a display or the like. This allows the user to specify one or more scene videos from the scene videos displayed in a list in a selectable manner for each scene type, and perform replacement processing on them. The display control unit 52 also controls the display of replacement videos on a display or the like. This allows the user to specify one replacement video from the replacement videos displayed in a list in a selectable manner, and output (play) it.

[0060] The base video image designation receiving unit 53 receives an operation to designate one base video image to be replaced from one or more videos provided by a video manufacturer. Specifically, the base video image designation receiving unit 53 receives an operation to designate one video image to be replaced from information indicating one or more videos displayed in a selectable manner on a display or the like of the output unit.

[0061] The replacement content designation receiving unit 54 receives an operation for designating the content of the replacement process. Specifically, the replacement content designation receiving unit 54 receives, as an operation for designating the content of the replacement process, an operation for designating by uploading a user image and an operation for designating the specific content of the replacement process.

[0062] The processing target designation receiving unit 55 receives designation of one or more scene videos to be subjected to the replacement process from among one or more scene videos presented to the user. For example, assume that five types of scene videos are extracted from a base video, including a daily life scene video, an introduction scene video, a development scene video, a battle scene video, and a conclusion scene video. In this case, the processing target designation receiving unit 55 receives designation of one or more scene videos to be subjected to the replacement process from among the extracted five types of scene videos. The processing target information described as being received by the processing target designation receiving unit 55 may also be received by the replacement content designation receiving unit as specific content of the replacement process. The designation of the processing target may include, for example, designation of one or more base video images to be subjected to the replacement process or multiple video images constituting a video consisting of a series of scenes.

[0063] The transmission control unit 56 controls the communication unit to transmit various types of information to the management server 1 and to the outside. For example, the transmission control unit 56 controls the communication unit to transmit, to the management server 1, operation information relating to an operation to specify one base video image to be replaced. The transmission control unit 56 also controls the communication unit to transmit, to the management server 1, operation information relating to an operation to specify the content of the replacement. The transmission control unit 56 also controls the communication unit to transmit, to the management server 1, information indicating that a predetermined operation (for example, an operation to press the BAD button or the SKIP button) was performed while the replacement video image was being output.

[0064] <Processing flow of Management Server 1> FIG. 4A is a flowchart showing an example of the flow of processing of the management server 1 up to the point where the user terminal 2 is caused to output the replaced moving image. The management server 1 acquires one or more videos transmitted from the video manufacturer server 3 (step S1) and presents the acquired one or more videos to the user in a selectable manner (step S2). When one of the one or more videos presented to the user is designated (YES in step S3), the management server 1 identifies the designated one video as the base video (step S4). On the other hand, when one video is not designated (NO in step S3), the management server 1 repeats the determination process of step S3.

[0065] The management server 1 extracts one or more video images of scenes from the base video image identified in step S4 (step S5), and associates each of the extracted one or more video images of scenes with a scene type (step S6).The management server 1 presents the one or more video images of scenes associated with the scene type to the user in a selectable manner (step S7).

[0066] When an operation to specify the content of the replacement process is performed on the user terminal 2 (YES in step S8), the management server 1 performs replacement process based on the specified content of the replacement process (step S9). As described above, the operation to specify the content of the replacement process includes an operation to specify the face of the user to be replaced by uploading a user image, and an operation to specify the specific content of the replacement process. On the other hand, when an operation to specify the content of the replacement process has not been performed (NO in step S8), the management server 1 repeats the determination process of step S8.

[0067] The management server 1 transmits the replaced video to the user terminal 2 (step S10). Then, the management server 1 causes the user terminal 2 to output the replaced video in a format that cannot be saved (step S11).

[0068] FIG. 4B is a flowchart showing an example of the flow of processing performed by the management server 1 while the replaced video is being output (played) on the user terminal 2. When a predetermined operation is performed while a replaced video is being output (played) on the user terminal 2 (YES in step S21), the management server 1 identifies the replaced video being output (played) when the operation was performed (step S22). On the other hand, when the predetermined operation is not performed (NO in step S21), the management server 1 repeats the determination process of step S21.

[0069] The management server 1 associates the replaced video identified in step S22 with information indicating that a predetermined operation was performed while the replaced video was being output (played) on the user terminal 2 (step S23).Then, the management server 1 feeds back information indicating the replaced video identified in step S22 to a predetermined AI model (step S24).

[0070] <Example> [Examples of this service] FIG. 5 is a diagram showing a specific example of this service. As shown in FIG. 5, in this service, videos V1 to Vm (m is an integer value equal to or greater than 1) that can be the target of replacement processing and that have been provided to service provider G by video manufacturer M are presented to user U. Specifically, information indicating each of videos V1 to Vm is displayed on user terminal 2 in a selectable manner. Note that videos V1 to Vm are each a series of Japanese period dramas, and are stored and managed in management server 1 (see FIG. 1). Here, it is assumed that the user operates user terminal 2 to specify video V1, a period drama set in the Warring States period.

[0071] In this service, scene videos Vs1 to Vsp (p is an integer value equal to or greater than 1) are extracted using a video V1 specified by a user as a base video. Then, a scene type is associated with each of the extracted scene videos Vs1 to Vsp. In the example of FIG. 5, it is assumed that the scene types are predetermined as one combination of scenes of daily life, introduction, development, battle, and conclusion. Therefore, each of the scene videos Vs1 to Vsp is associated with one of scenes of daily life, introduction, development, battle, and conclusion.

[0072] Here, as a simple example, it is assumed that scene videos Vs1 to Vs5 of scene videos Vs1 to Vsp extracted from video V1 are associated with each other. Specifically, it is assumed that scene video Vs1 is an everyday scene (3 minutes), scene video Vs2 is an introductory scene (3 minutes), scene video Vs3 is an unfolding scene (3 minutes), scene video Vs4 is a battle scene (5 minutes), and scene video Vs5 is a conclusion scene (3 minutes). Note that the times in parentheses indicate the playback time of the video of each scene.

[0073] In this service, scene videos Vs1 to Vs5, each associated with a scene type, are presented to the user U. Specifically, information indicating each of the scene videos Vs1 to Vs5 is displayed on the user terminal 2 in a selectable manner. The user then operates the user terminal 2 to specify the content of the replacement process. Here, for example, suppose the user specifies all of the scene videos Vs1 to Vs5 as a series of videos to be subjected to the replacement process. Furthermore, suppose the user specifies a process to replace the face of the main character, who is a character in the scene videos Vs1 to Vs5, with the user's face and uploads user image data. The management server 1 then simultaneously performs the replacement process on each of the scene videos Vs1 to Vs5.

[0074] This service allows videos Vs1 to Vs5 of scenes that have been replaced to be output (played back) from the user terminal 2 as replacement videos. The user can save the videos Vs1 to Vs5 of scenes that have been replaced in a playlist as favorite replacement videos. However, the user terminal 2 is controlled so that operations to save the replacement videos are not accepted. A playlist is a list that specifies the playback order of two or more videos, including videos that have been replaced. A playlist may include a list selected by a user or a service provider to create a storyline for multiple scenes extracted from multiple base videos. A playlist may also include videos that have been replaced but have not had scenes extracted. A playlist may also include base videos that have not been replaced. A playlist is generated by the processing target designation receiving unit 55 in conjunction with the received content of the designation of the processing target, but the content of the playlist may be changed after the replacement process is completed.

[0075] Here, the management server 1 can extract scenes from other videos V2 in the same way as the videos Vs1 to Vs5 of scenes extracted from the video V1 described above, and manage them in association with the scene types. Furthermore, the management server 1 can manage multiple types of scenes that are output consecutively as a series of scene types. Specifically, for example, the following types of series of scenes are managed as types of series of scenes: everyday scenes, introduction scenes, development scenes, battle scenes, and conclusion scenes. The management server 1 can manage multiple sets of series of scene types by grouping the types of series of scenes together. In other words, it can be said that the management server 1 can manage the composition of a video from a series of scenes. For example, when a user selects a series of scene types (sets) such as daily life scenes, introduction scenes, development scenes, battle scenes, and conclusion scenes, the management server 1 presents the user with video images corresponding to each scene type as candidates. The user then selects a video image for each scene type. This allows a series of replacement video images to be created by combining multiple video images (e.g., video images V1 and V2).

[0076] Furthermore, the management server 1 can evaluate the accuracy of each of the extracted scene videos in advance when a replacement process is performed, and associate and manage the evaluation values. Furthermore, the management server 1 can display only those scene types with a high evaluation value of the accuracy when a replacement process is performed as candidates for scene videos that can be used in the early part (e.g., the first half) of a series of scene types. In other words, those scene types with a low evaluation value of the accuracy when a replacement process is performed are presented only in the later part (e.g., the second half) of a series of scene types. This ensures that when a user plays back the replaced video, it is possible to ensure a long time for the replacement process for those scene types with a low evaluation value of the accuracy when a replacement process is performed. Furthermore, the user will view the replaced video, viewing the early part with high accuracy, before viewing the later scenes. This has the effect of making the user less likely to notice those scene types with a low evaluation value of the accuracy when a replacement process is performed after viewing the early scenes and immersing themselves in the replaced video.

[0077] [Specific example of replacement processing] FIG. 6 is a diagram showing a specific example of the replacement process. 5, the replacement process for the scene videos Vs1 to Vs5 involves replacing the face of the main character C, who is a character in the scene videos Vs1 to Vs5, with the face of the user. Specifically, as shown in Fig. 6, the face portion Cf of the main character C appearing in each of the scene videos Vs1 to Vs5 is replaced with the face portion Uf of the user U included in the user image uploaded from the user terminal 2.

[0078] <Advantageous Effects of the Present Embodiment> According to the above-described embodiment, it is possible to provide a mechanism that enables a user to create a replaced video that the user can enjoy privately with simpler operations.

[0079] <Other> Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications and improvements within the scope of achieving the object of the present invention are included in the present invention. For example, the specific examples shown in Figures 5 and 6 are merely examples for achieving the object of the present invention, and are not particularly limited.

[0080] Furthermore, the above-described series of processes can be executed by hardware or software. In other words, the above-described functional configuration is merely an example and is not particularly limited. In other words, it is sufficient for the information processing system to be provided with a function that can execute the above-described series of processes as a whole, and the type of functional block used to realize this function is not particularly limited to the above-described example.

[0081] In the above-described embodiment, the replacement process involves replacing the facial portion of the main character appearing in one or more scenes of video with the facial portion of the user included in a user image uploaded by the user, but is not limited to this. For example, instead of uploading a user image, an operation to input a description of the face to be replaced (for example, a description of facial features preferred by the user) may be accepted, and an AI may be made to generate a facial image based on the input information.

[0082] In the above-described embodiment, the process of associating a scene type with each of the videos of one or more scenes is configured to be performed automatically by AI or the like, but is not limited to this. For example, a user may perform an operation to associate a scene type with each of the videos of one or more scenes.

[0083] Furthermore, in the above-described embodiment, when a base moving image is specified from one or more moving images presented to a user, the information processing system S extracts moving images of one or more scenes from the specified base moving image, but this is not particularly limited to this.

[0084] For example, the information processing system S (e.g., the extraction unit 34) preliminarily extracts one or more scene videos from a base video (not selected by the user) for one or more videos to be presented to the user. Then, the information processing system S (e.g., the association unit 34) associates a scene type with each of the one or more scene videos extracted by the extraction unit 33. Thereafter, the information processing system S (e.g., the display control unit 52) ​​presents one or more videos to the user in a selectable manner. At this time, the user can select one or more videos for each scene type. That is, the information processing system S (e.g., the acquisition unit 31) acquires the content of the replacement process, including the selection of one or more selected videos associated with the scene type specified by the user. Then, the information processing system S (e.g., the replacement unit 35) performs the replacement process based on the content of the replacement process specified by the user. Other processes are basically the same as those in the above-described embodiment, but may be modified as appropriate. In this way, the user can select and combine one or more videos from the videos of one or more scenes extracted in advance, and view them as a series of videos that have been subjected to replacement processing.

[0085] The location of the functional blocks is not particularly limited and may be arbitrary. For example, the functional blocks of the management server 1 may be transferred to another device, or the functional blocks of another device may be transferred to a server. Furthermore, one functional block may be configured as a single piece of hardware, a single piece of software, or a combination of both.

[0086] When a series of processes is executed by software, the programs constituting the software are installed onto a computer or the like from a network or a recording medium. The computer may be a computer incorporated into dedicated hardware. The computer may also be a computer capable of executing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0087] The recording medium containing such a program may be configured as a removable medium (not shown) that is distributed separately from the device main body in order to provide the program to users, etc., or may be configured as a recording medium that is pre-installed in the device main body and provided to users, etc. Since the program can be distributed via a network, the recording medium may be installed in or accessible from a computer that is connected or connectable to the network.

[0088] In this specification, the steps describing the program recorded on the recording medium include not only processes that are performed in chronological order, but also processes that are not necessarily performed in chronological order but are performed in parallel or individually. Also, in this specification, the term "system" means an overall device composed of multiple devices or multiple means, etc.

[0089] In other words, the information processing system to which the present invention is applied can take various forms having the following configurations. (1) That is, the information processing system S to which the present invention is applied includes a replacement target object designation receiving means (e.g., replacement target object designation receiving unit 55 in FIG. 3) that receives designation of a target object to be replaced among moving images (e.g., designation of a base moving image or a moving image of a scene to be replaced, designation of a character in the moving image of a scene, etc.); a replacement content specification receiving unit (e.g., the replacement target object specification receiving unit 54 in FIG. 3) that receives a specification of a post-replacement object (e.g., specification of a user's face included in the user image data) to be replaced from the target object by the replacement process; a replacement means (for example, the replacement unit 35 in FIG. 3) for performing a replacement process between the designated target object and the replaced object for the designated video; and an output control means (for example, the output control unit 36 ​​in FIG. 3) that outputs the moving image on which the replacement process has been performed as a replaced moving image.

[0090] This allows the user to simply specify the content of the replacement process for the video, and the replaced video after the replacement process is output (e.g., played back).As a result, the user can create a replaced video that can be enjoyed privately with simple operations.

[0091] (2) The apparatus may further include an extraction means for extracting one or more scene videos from a base video image, wherein the replacement target object designation receiving means receives the designation of a target object to be subjected to the replacement process for the extracted scene video image, the replacement means performs a replacement process between the designated target object and the replaced object for the designated scene video image, and the output control means outputs the video image consisting of the plurality of scenes subjected to the replacement process as a replaced video image.

[0092] This allows a user to simply specify the content of replacement processing for one or more scenes extracted from a base video for each scene type, and the replaced video is output (e.g., played back). As a result, the user can create a replaced video that can be enjoyed privately with simple operations. Furthermore, if the user specifies multiple scenes, the user can enjoy a video consisting of a series of scenes as the replaced video.

[0093] (3) The replacement means can also perform the replacement process on each of the designated plurality of moving images of the scenes simultaneously in parallel. Since multiple videos of scenes with a relatively short playback time of about 3 to 5 minutes are extracted, the load does not increase even if the replacement process is performed simultaneously in parallel. As a result, efficient replacement process can be realized. Furthermore, when a user performs cue playback for each scene, even if there are videos of scenes for which replacement process has not been completed, the videos can be played in order starting from the videos of scenes for which replacement process has been completed. In other words, the videos can be played in order starting from the videos of scenes for which replacement process has been completed without waiting for the completion of replacement process for all scenes.

[0094] (4) The apparatus further includes a matching means (e.g., the matching unit 34 in FIG. 3) for matching and managing a scene type (e.g., daily life, introduction, development, battle, conclusion, etc.) for each of the extracted moving images of the scene, and the matching means can select one scene type from a combination of a plurality of predetermined scene types and match it with the moving image of the scene. This allows for efficient replacement processing.

[0095] (5) Furthermore, the replacement means can provide a difference in the accuracy of the replacement process for each of the multiple types of moving images of the scenes that are output consecutively, between those that are output first and those that are output later. This allows, for example, the accuracy of the replacement process for the first output (played back) of multiple types of video scenes to be higher than the accuracy of the replacement process for the later output (played back) of the video. In this case, for example, a user viewing a video of a first half scene with a high accuracy of the replacement process is more likely to empathize. In contrast, since the user will be viewing a video of a second half scene in a state of empathy, the effect is that the user is less likely to feel uncomfortable even if the accuracy of the replacement process is lowered to a certain extent. In particular, if there is a battle scene towards the end of the story of the base video, the movements of the characters become more intense and the movement of the characters' faces also becomes faster, making it difficult to increase the accuracy of the replacement process. This results in a high effect.

[0096] (6) The extraction means can also extract a moving image of the scene based on the aspect (e.g., direction, speed of movement, etc.) of a part (e.g., a face of a character) contained in the base moving image that may be the subject of the replacement process. This allows for the preferential extraction of video of scenes that are easy to replace, such as when the face of a character is a potential target for replacement, such as when the face of a character faces forward or when the speed at which the face of a character moves is slow.

[0097] (7) The output control means can also cause the terminal that outputs the replaced video to output the replaced video in a manner that prevents the replaced video from being saved. This prevents replacement videos created using this service from being spread as fake videos on social networking services (SNS), etc. As a result, users can enjoy replacement videos privately without worry.

[0098] (8) Furthermore, when a predetermined operation (for example, pressing a BAD button or a skip button) is performed while the replacement video is being output (played), the device may further include a management means (for example, information management unit 32 in FIG. 3) for identifying and managing the replacement video being output when the operation is performed. This allows the user to point out the low accuracy of the replacement process by performing a predetermined operation while viewing the replacement video. Also, if the replacement video is available to a service provider providing this service, the service provider can also point out the low accuracy of the replacement process by performing a predetermined operation while viewing the replacement video.

[0099] (9) The management means can further identify and manage one or more frames that constitute the replaced video that is being output when the operation is performed. This allows the user to perform a predetermined operation while viewing the replacement video, thereby indicating that the accuracy of the replacement process was low at the time the predetermined operation was performed. Furthermore, if the replacement video can be viewed by a service provider providing this service, the service provider can also perform a predetermined operation while viewing the replacement video, thereby indicating that the accuracy of the replacement process was low at the time the predetermined operation was performed.

[0100] (10) Furthermore, the information processing device to which the present invention is applied can take various forms having the following configurations. In other words, the management server 1 as an information processing device to which the present invention is applied is an information processing device having an extraction means for extracting moving images of one or more scenes from a base moving image, an association means for associating a scene type with each of the extracted moving images of the scenes, a replacement content designation receiving means for accepting designation of the content of the replacement process, a processing target designation receiving means for accepting designation of the target of the replacement process among the extracted moving images of the scenes, a replacement means for performing the designated replacement process on the moving images of the designated scenes, and an output control means for outputting the moving images of the scenes after the replacement process as replaced moving images.

[0101] (11) Furthermore, the information processing method to which the present invention is applied can take various forms having the following configurations. In other words, the information processing method to which the present invention is applied is an information processing method including the steps of: accepting the designation of a target object in a moving image that is to be subjected to a replacement process; accepting the designation of a replaced object to be replaced from the target object by the replacement process; performing the replacement process between the designated target object and the replaced object for the designated moving image; and outputting the moving image after the replacement process as a replaced moving image.

[0102] (12) Furthermore, the program to which the present invention is applied can take various forms having the following configurations. That is, the program to which the present invention is applied is a program that causes an information processing system (for example, information processing system S in FIG. 1) to execute information processing including the steps of accepting the designation of a target object in a moving image that is to be subjected to a replacement process, accepting the designation of a replaced object to be replaced from the target object by the replacement process, performing the replacement process between the designated target object and the replaced object for the designated moving image, and outputting the moving image after the replacement process as a replaced moving image. [Explanation of symbols]

[0103] 1: Management server, 2: User terminal, 3: Video manufacturer server, 11: CPU, 16: Output unit, 17: Input unit, 18: Storage unit, 19: Communication unit, 31: Acquisition unit, 32: Information management unit, 33: Extraction unit, 34: Correlation unit, 35: Replacement unit, 36: Output control unit, 51: Acquisition unit, 52: Display control unit, 53: Base video designation acceptance unit, 54: Replacement content designation acceptance unit, 55: Processing target designation acceptance unit, 56: Transmission control unit, S: Information processing system, N: Network

Claims

1. a replacement target object designation receiving means for receiving designation of a target object to be subjected to replacement processing in the moving image; a replacement content designation receiving means for receiving a designation of a post-replacement object to be replaced from the target object by the replacement process; a replacement means for performing a replacement process between the designated target object and the replaced object for the designated video; an output control means for outputting the moving image subjected to the replacement process as a replaced moving image; An information processing system having the above.

2. an extraction means for extracting one or more scenes from the base moving image; Furthermore, The replacement target object designation receiving means Accepting designation of a target object to be subjected to replacement processing for the extracted moving image of the scene; The replacing means performing a replacement process of the designated target object with the replaced object for the moving image of the designated scene; The output control means outputting the moving image made up of the plurality of scenes subjected to the replacement process as a replaced moving image; The information processing system according to claim 1 .

3. the replacement means performs the replacement process on each of the specified plurality of moving images of the scenes simultaneously in parallel. The information processing system according to claim 2 .

4. a correlation means for correlating each of the extracted scene moving images with a scene type and managing the scene moving images; Further provided with the associating means selects one scene type from a combination of a plurality of predetermined scene types and associates the selected scene type with the moving image of the scene; The information processing system according to claim 2 .

5. the replacement means performs the replacement process with a higher accuracy on an earlier output of the plurality of types of moving images of the scenes that are successively output than on an earlier output of the moving images of the scenes that are successively output. The information processing system according to claim 4 .

6. the extraction means extracts the moving image of the scene based on a feature of a portion included in the base moving image that can be a target of the replacement process; The information processing system according to claim 2 .

7. the output control means outputs the replaced video in a manner that the replaced video cannot be saved on the terminal that outputs the replaced video. The information processing system according to claim 2 .

8. The video output device further comprises a management unit that, when a predetermined operation is performed while the replaced video is being output, identifies and manages the replaced video being output when the operation is performed. The information processing system according to claim 2 .

9. the management means further identifies and manages one or more frames constituting the replaced moving image being output when the operation is performed. The information processing system according to claim 8 .

10. receiving a designation of a target object to be replaced from within the video; receiving a designation of a post-replacement object to be replaced by the replacement process from the target object; performing the replacement process between the designated target object and the replaced object for a designated video; outputting the moving image subjected to the replacement process as a replaced moving image; An information processing method including:

11. Information processing systems, receiving a designation of a target object to be replaced from within the video; receiving a designation of a post-replacement object to be replaced by the replacement process from the target object; performing the replacement process between the designated target object and the replaced object for a designated video; outputting the moving image subjected to the replacement process as a replaced moving image; A program for executing control processing including:

Images