Content configuration method and content configuration device

Abstract

According to the present invention, a person portion (person image) included in video content (moving image) is extracted, and on the basis of a representation mode [live-action, 3D CG, illustration, schematic figure, or no drawing] of the person image which is determined such that a level of detail decreases from [high] to [low] in accordance with, for example, a viewing time (operation time t) that is reset for each scene of the moving image (or for each utterance of a user), the extracted person portion (person image) is displayed while being converted into the person image in the determined representation mode. Similarly, an audio corresponding to the extracted person portion is also separated from the moving image and output after being converted into [three-dimensional sound, stereo, monaural, or no sound].

Description

Content composition method and content composition device

[0001] FIELD OF THE INVENTION Embodiments of the present invention relate to a content composition method and apparatus.

[0002] For example, there are video services available that aim to give users a simulated experience of an environment different from the actual environment in which they exist, using video content, such as immersive games that use VR (Virtual Reality) goggles to give users the feeling of being immersed in the game, and simulated travel services that allow users to experience a simulated trip while watching a high-definition television monitor at home.

[0003] These video services feature characters that users encounter in games, tour guides, etc. When displaying such characters in content, rather than simply displaying pre-filmed images of the characters, for example, by displaying them while rendering 3DCG (3-dimensional computer graphics) in accordance with the user's speech and actions, this has the effect of naturally portraying communication between the user and the characters, thereby further enhancing the sense of realism of the service.

[0004] Japanese Patent Publication No. 2020-194569 Japanese Patent Publication No. 2019-220014

[0005] However, 3DCG rendering places a high load on general-purpose IT devices, and depending on the user's environment, problems such as poor display of 3DCG or processing delays can occur. For this reason, it can only be used in limited high-spec environments, and even if it is used, services that are intended to enhance the sense of realism can actually make the image seem unnatural.

[0006] The present invention has been made in consideration of such problems, and aims to provide a content composition method and a content composition device that make it possible to compose video content that provides a user with a high sense of realism without requiring high-load processing.

[0007] The content configuration method of the present invention causes a processor to output video content to a user's output device, acquire the user's operation time associated with the output of the video content, determine a level of detail corresponding to the acquired user's operation time based on a table in which multiple levels of detail are set so that the level of detail for expressing the video content decreases depending on the operation time, convert the expression form of the person part extracted from the video content into an expression form with the determined level of detail, and reconstruct the video content.

[0008] According to the present invention, it is possible to create video content that gives a user a sense of realism without requiring high-load processing.

[0009] FIG. 1 is a block diagram showing an example of the functional configuration of a content composition device 100 according to an embodiment of the present invention. FIG. 2 is a flowchart showing content composition setting processing according to the function of the content composition setting unit 13 in the content composition device 100. FIG. 3 is a diagram showing a t table referenced based on the operation time t of video content in the content composition device 100's content composition setting processing. (A) of FIG. 3 shows a visual setting table Tv that defines different visual setting information (image expression form) tv depending on the operation time t, and (B) of FIG. 3 shows an auditory setting table Th that defines different auditory setting information (audio expression form) th depending on the operation time t. FIG. 4 is a flowchart showing visual information reconstructing processing according to the function of the visual information reconstructing unit 14 in the content composition device 100. FIG. 5 is a flowchart showing auditory information reconstructing processing according to the function of the auditory information reconstructing unit 15 in the content composition device 100. FIG. 6 is a flowchart showing content reconstructing processing according to the function of the content reconstructing unit 16 in the content composition device 100. FIG. 7 is a block diagram showing an example of the hardware configuration of the content composition device 100 according to an embodiment of the present invention.

[0010] Hereinafter, an embodiment of a content composition method and a content composition device of the present invention will be described with reference to the drawings.

[0011] (Configuration of the embodiment) FIG. 1 is a block diagram showing an example of the functional configuration of a content composition apparatus 100 according to an embodiment of the present invention.

[0012] The content composition device 100 is realized in an information processing device such as a server device, a PC (Personal Computer), a tablet terminal, or smart glasses equipped with a processor (CPU: Central Processing Unit) by having the processor execute the functions of each part shown in FIG. 1 in accordance with a program (through cooperation between hardware and software).

[0013] The content composition device 100 includes a content presenting unit 11 , an operation time acquiring unit 12 , a content composition setting unit 13 , a visual information reconstructing unit 14 , an auditory information reconstructing unit 15 , and a content reconstructing unit 16 .

[0014] The content presenting unit 11 displays (outputs) video content (e.g., video including audio) on a display device used by a user. The video content may be provided by a video service used via a communication network such as the Internet, or may be recorded on a recording medium such as a hard disc drive (HDD), a solid state drive (SSD), or a digital versatile disc (DVD).

[0015] The operation time acquisition unit 12 acquires and stores the operation time (viewing time) t during which the user uses the video content displayed by the content presentation unit 11. The operation time t of the video content is passed to the content configuration setting unit 13 and used in processing by the content configuration setting unit 13.

[0016] The operation time t may be reset to "0" when the scene of the video content changes. The operation time t may also be reset to "0" when a user utterance event occurs. The acquisition of information on the scene change of the video content and information on the user utterance accompanying a dialogue event of the video content is performed by the content reconstructing unit 16 described later.

[0017] The content configuration setting unit 13 determines the configuration form of the video content (including the image representation form and the audio representation form) based on the operation time t received from the operation time acquisition unit 12. The configuration form of the video content determined by the content configuration setting unit 13 is passed to the visual information reconstructing unit 14 and the auditory information reconstructing unit 15 and used for processing therein. The function of the content configuration setting unit 13 will be described in detail later.

[0018] The visual information reconstructor 14 reconstructs the visual information based on the configuration form (image representation form) of the video content received from the content configuration setting unit 13. The visual information reconstructed here is passed to the content reconstructor 16 and used in processing by the content reconstructor 16. The function of the visual information reconstructor 14 will be described in detail later.

[0019] The auditory information reconstructor 15 reconstructs the auditory information based on the configuration form (audio expression form) of the video content received from the content configuration setting unit 13. The auditory information reconstructed here is passed to the content reconstructor 16 and used in processing by the content reconstructor 16. The function of the auditory information reconstructor 15 will be described in detail later.

[0020] The content reconstructor 16 reconstructs the video content using the visual information and audio information received from the visual information reconstructor 14 and the audio information reconstructor 15. The content reconstructor 16 also monitors scene information and dialogue events of the video content being displayed, and continues to determine the configuration form of the video content (including the image representation form and the audio representation form) based on the operation time t, reconstruct the visual information and audio information based on the determined configuration form of the video content, and reconstruct the video content using the reconstructed visual information and audio information, until the scene changes or the user speaks.

[0021] When the content reconstructing unit 16 detects a scene change, it passes that information (scene change information) to the operation time acquiring unit 12. Furthermore, when the content reconstructing unit 16 detects a user utterance accompanying a dialogue event, it passes that information (utterance information) to the operation time acquiring unit 12.

[0022] The video content reconstructed by the content reconstruction unit 16 is sent to the content presentation unit 11 and displayed (output) on the display device used by the user. The function of the content reconstruction unit 16 will be described in detail later.

[0023] (Operation of the embodiment) Next, the operation of the content composition apparatus 100 of the embodiment will be described.

[0024] When the content presentation unit 11 starts displaying (outputting) the video content (video) provided to the content composition device 100, the operation time acquisition unit 12 first acquires the operation time t (e.g., equivalent to the user's viewing time) from the start of the display of the video content.

[0025] Here, it is assumed that the video content to be displayed includes an image (person image) of a person, for example, who is guiding the content of the video content or who is interacting with the user, photographed together with the video content or synthesized into the video content.

[0026] FIG. 2 is a flowchart showing a content configuration setting process according to the function of the content configuration setting unit 13 in the content configuration device 100.

[0027] Figure 3 shows tables t that are referenced based on the operation time t of the video content in the content configuration setting process of the content configuration device 100, where (A) shows a visual setting table Tv that defines different visual setting information (image expression form) tv depending on the operation time t, and (B) shows an auditory setting table Th that defines different auditory setting information (audio expression form) th depending on the operation time t.

[0028] When the content configuration setting unit 13 receives the operation time t acquired by the operation time acquisition unit 12 (step S11), it refers to the visual setting table Tv (first table) as shown in Figure 3 (A) to determine visual setting information (image expression form) tv corresponding to the operation time t, and also refers to the auditory setting table Th (second table) as shown in Figure 3 (B) to determine auditory setting information (audio expression form) th corresponding to the operation time t (step S12).

[0029] In the visual setting table Tv shown in FIG. 3(A), for example, the options for the visual setting information (image expression format) tv are set as tv1 [live action], tv2 [3DCG], tv3 [illustration], tv4 [schematic diagram], and tv5 [no drawing of people] in correspondence with the step-by-step ranges tv1, tv2, ..., tv5 of the operation time t.

[0030] In the auditory setting table Th shown in FIG. 3(B), for example, the options for auditory setting information (audio expression form) th are set as th1 [stereophonic sound], th2 [stereo], th3 [monaural], and th4 [no human voice] in correspondence with the step-wise ranges th1, th2, th3, and th4 of the operation time t.

[0031] Here, if the operation time t is, for example, "130", the visual setting information (image expression format) tv is determined to be [illustration] corresponding to the value range tv3 of the visual setting table Tv, and the person image displayed in the video content will be expressed as an illustration. Similarly, if the operation time t is, for example, "130", the auditory setting information (audio expression format) th is determined to be [stereo] corresponding to the value range th2 of the auditory setting table Th, and the audio of the person image displayed in the video content will be expressed in stereo.

[0032] The contents of the t table (visual setting table Tv and auditory setting table Th) may be freely edited and set as desired by, for example, a service provider that provides video content, in accordance with the content and interaction of the video content.

[0033] The basis for setting such a t table is the idea in the field of cognitive psychology that "after a certain degree of 'imprinting' has been done on the human brain, even if some of the information is lost, the brain can supplement and recognize that information." Therefore, even if the visual setting information (image expression format) tv and the auditory setting information (audio expression format) th, which are information for expressing the characters (person images) in the video content, change from information for detailed expression (rich) to information for simple expression (poor) as the operation time t (equivalent to the viewing time of the video content) passes, that is, as the service usage time passes, the user can still recognize that the same characters are present in the video content they are watching.

[0034] FIG. 4 is a flowchart showing the visual information reconstructing process according to the function of the visual information reconstructing unit 14 in the content composition apparatus 100. As shown in FIG.

[0035] When the visual information reconstructing unit 14 receives the visual setting information tv determined by the content configuration setting unit 13 (step S21), it extracts the person portion to be processed from the video content (moving image) being displayed (step S22).

[0036] In this case, image analysis (object analysis) may be used to extract the person portion, or the user may manually set the person portion. Furthermore, extraction of the person portion may be performed for each frame of the video, or may be performed once every arbitrary number of frames to reduce the processing load.

[0037] Based on the visual setting information tv received in step S21 and the information on the person parts extracted in step S22 (including area information on the person parts in the video and information on the person images), the visual information reconstruction unit 14 generates visual (person) information by converting (changing) the expression form of the person images of the extracted person parts in accordance with the determined visual setting information tv (step S23).

[0038] For example, if the visual setting information tv determined by the content configuration setting unit 13 is "live action," the person image in the video is used as is. Also, if the visual setting information tv determined by the content configuration setting unit 13 is "illustration," an illustration of the person is generated to match the extracted person image.

[0039] Here, as a method for generating an illustration of a person, for example, an illustration-like image may be automatically generated by extracting the feature amount of a person image, or an illustration may be drawn and imported as an electronic file.

[0040] The visual information reconstructor 14 passes the information on the person portion extracted in step S22 (area information on the person portion in the video) together with the visual (person) information generated in step S23 to the content reconstructor 16 (step S24).

[0041] FIG. 5 is a flowchart showing the auditory information reconstructing process in accordance with the function of the auditory information reconstructing unit 15 in the content composition apparatus 100. As shown in FIG.

[0042] When the auditory information reconstruction unit 15 receives the auditory setting information th determined by the content configuration setting unit 13 (step S31), it separates the audio (audio corresponding to the extracted person part) from the video content (video) being displayed (step S32).

[0043] The auditory information reconstructing unit 15 generates auditory (audio) information by converting (changing) the audio separated in step S32 in accordance with the auditory setting information th determined by the content configuration setting unit 13 (step S33).

[0044] For example, if the auditory setting information th determined by the content configuration setting unit 13 is [stereophonic sound], auditory (audio) information is generated in which the separated audio is converted into stereophonic sound, and if the determined auditory setting information th is [monaural], auditory (audio) information is generated in which the separated audio is converted into monophonic sound.

[0045] The auditory information reconstructing unit 15 passes the auditory (audio) information generated in step S33 to the content reconstructing unit 16 (step S34).

[0046] FIG. 6 is a flowchart showing a content reconstructing process in accordance with the function of the content reconstructing section 16 in the content constructing apparatus 100. As shown in FIG.

[0047] The content reconstructor 16 receives the visual information (including area information of the person part in the video and the person image whose expression form has been converted according to the operation time t) generated by the visual information reconstructor 14 and the auditory information (including the sound whose expression form has been converted according to the operation time t) generated by the auditory information reconstructor 15 (step S41), and then synthesizes the generated visual information and auditory information with the video content (video) being displayed to reconstruct the video content (video) (step S42).

[0048] That is, the content reconstructor 16 reconstructs the video content (video) by synthesizing, for the video content (video) being displayed, a person image whose expression form has been converted based on area information of the person part in the video contained in the generated visual information, and by synthesizing, in place of the audio of the video content (video) being displayed, audio whose expression form has been converted and contained in the generated auditory information. The reconstructed video content is passed to the content presenter 11 (step S43), and is displayed (output) on the user's display device.

[0049] Furthermore, for example, in parallel with the process of synthesizing visual information and audio information into video content (video) (step S42), the content reconstructor 16 extracts and analyzes scene information from the video (step S44) and detects scene changes (step S45). When a scene change is detected (step S45 (YES)), the content reconstructor 16 passes the scene change information to the operation time acquirer 12 (step S46). The operation time acquirer 12 resets the operation time t to "0" in accordance with the scene change information and starts acquiring a new operation time t.

[0050] Furthermore, for example, in parallel with the process of synthesizing visual information and auditory information into video content (video) (step S42), the content reconstructor 16 extracts and analyzes dialogue events from the video (step S47) and detects user utterances input from a voice input device used by the user in the interactive video content (video) (step S48). When the user utterance is detected (step S48 (YES)), the content reconstructor 16 passes the utterance information representing the detected utterance to the operation time acquirer 12 (step S49). The operation time acquirer 12 resets the operation time t to "0" in accordance with the utterance information and starts acquiring a new operation time t.

[0051] When the operation time t acquired by the operation time acquisition unit 12 is reset to "0" and acquisition of a new operation time t begins, the following processes are repeatedly performed, as described above: determining the visual setting information (expression form of a person's image) and the auditory setting information (expression form of an audio) according to the newly acquired operation time (viewing time) t (steps S11 and S12); extracting and converting (generating) the image of a person in the video content (video) according to the determined visual setting information (expression form of a person's image) (steps S21 to S24); separating and converting (generating) the audio in the video content (video) according to the determined audio setting information (expression form of an audio) (steps S31 to S34); synthesizing the generated visual information (image of a person) and audio information (audio) with the video content (video) (steps S41 to S43); extracting and detecting scene information (steps S44 to S46); and extracting and detecting dialogue events (steps S47 to S49).

[0052] FIG. 7 is a block diagram showing an example of the hardware configuration of content composition apparatus 100 according to an embodiment of the present invention.

[0053] 7, content composition apparatus 100 according to the embodiment is configured, for example, by a server computer or a personal computer, and includes a hardware processor 111A such as a CPU (Central Processing Unit). Program memory 111B, data memory 112, input / output interface 113, and communication interface 114 are connected to this hardware processor 111A via bus 115.

[0054] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network NW. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0055] An input device 200 and an output device 300 that are attached to the content composition apparatus 100 and used by a user or the like are connected to the input / output interface 113 .

[0056] The input / output interface 113 takes in operation data and voice data input by a user or the like through an input device 200 such as a keyboard, touch panel, touchpad, mouse, or microphone, and outputs the output data to an output device 300, which may include a display device using a liquid crystal or organic electroluminescence (EL) display, or a voice output device using a speaker, for display or voice output. Note that the input device 200 and the output device 300 may be devices built into the content composition apparatus 100, or may be input devices and output devices of another information processing apparatus (information processing terminal) that can communicate with the content composition apparatus 100 via the network NW.

[0057] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and stores the programs necessary to execute various control processes according to the embodiment.

[0058] The data memory 112 is a tangible storage medium that is a combination of, for example, the nonvolatile memory described above and a volatile memory such as RAM (Random Access Memory), and is used to store various data acquired and generated during various processes.

[0059] The content composition device 100 according to an embodiment of the present invention can be configured as a data processing device having, as software processing function units, a content presentation unit 11, an operation time acquisition unit 12, a content composition setting unit 13, a visual information reconstruction unit 14, an auditory information reconstruction unit 15, and a content reconstruction unit 16 shown in FIG. 1.

[0060] Each information storage unit used as a working memory by each unit of content composition apparatus 100 may be configured using data memory 112 shown in Fig. 7. However, these configured storage areas are not essential components within content composition apparatus 100, and may be areas provided in an external storage medium such as a USB (Universal Serial Bus) memory, or in a storage device such as a database server located in the cloud.

[0061] The processing function units in each of the aforementioned content presenting unit 11, operation time acquiring unit 12, content configuration setting unit 13, visual information reconstructing unit 14, auditory information reconstructing unit 15, and content reconstructing unit 16 can all be realized by reading and executing a program stored in program memory 111B by hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as application specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0062] According to the content composition device 100 of the embodiment, instead of displaying a person's image appearing in video content (moving image) viewed by a user using a heavy processing load, such as 3DCG rendering, the content composition device 100 extracts a person's portion (person's image) from the video, and converts the extracted person's portion (person's image) into a person's image of a determined representation format (live action / 3DCG / illustration / schematic / no drawing) so that the level of detail varies from "high" to "low" depending on the viewing time (operation time t) that is reset for each scene of the video (or each time the user speaks). Similarly, the audio corresponding to the extracted person's portion is separated from the video and converted (stereophonic / stereo / monaural / no audio) for output.

[0063] For example, if one scene of a video is long, the image of the person extracted from the video is gradually changed from a high level of image detail (live action) to a low level of image detail (no drawing), and the audio corresponding to the person separated from the video is gradually changed from a high level of audio detail (stereophonic sound) to a low level of audio detail (no audio).

[0064] In this way, the content composition device 100 of the embodiment can realize a video content composition method for realizing a video service that allows users to feel the presence of others while reducing the processing load by dynamically changing the content composition while applying the mechanism by which humans sense the presence of others who are acting together, based on findings from cognitive psychology. Therefore, it is possible to compose video content that gives users a high sense of realism without requiring high-load processing.

[0065] In the above-described embodiment, when the representation format of a person image, which is the visual setting information determined by the content configuration setting unit 13, is tv5 [no person drawn], for example, the person image of the person portion extracted from the video is erased and the background image is complemented. In this case, the background image may be complemented by, for example, copying from another frame of the video, generating using AI, or any other method.

[0066] Furthermore, in the above-described embodiment, when the video content to be configured is a moving image with interaction, the content configuration setting unit 13 performs processing by excluding the visual setting information tv5 [no drawing of a person] and the auditory setting information tv4 [no audio of a person], which are options in the t table (visual setting table Tv and auditory setting table Th).

[0067] The techniques described in each of the above-described embodiments can be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (a floppy disk, a hard disk, etc.), an optical disc (a CD-ROM, a DVD, an MO, etc.), a semiconductor memory (a ROM, a RAM, a flash memory, etc.), or the like, and can be transmitted and distributed via a communication medium.

[0068] The program stored on the medium also includes a configuration program that configures the software means (including not only the execution program but also tables and data structures) that the computer executes. The computer that implements this device reads the program recorded on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-mentioned processing by having its operation controlled by this software means. Note that the recording medium referred to in this specification is not limited to storage media for distribution, but also includes storage media such as magnetic disks and semiconductor memories installed inside the computer or in devices connected via a network.

[0069] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by selecting and combining the multiple disclosed constituent elements.

[0070] For example, if the problem can be solved and an effect can be obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiment, the configuration from which these constituent elements are deleted can be extracted as an invention.

[0071] DESCRIPTION OF SYMBOLS 100...Content composition device 11...Content presentation unit 12...Operation time acquisition unit 13...Content composition setting unit 14...Visual information reconstructing unit 15...Auditory information reconstructing unit 16...Content reconstructing unit Tv...Visual setting table Th...Auditory setting table 111A...Processor 111B...Program memory 112...Data memory 200...Input device 300...Output device NW...Communication network

Claims

1. A content construction method that causes a processor to execute the following processes: output video content to a user's output device; acquire the user's operation time associated with the output of the video content; determine a level of detail corresponding to the acquired user's operation time based on a table in which multiple levels of detail are set so that the level of detail for expressing the video content decreases depending on the operation time; convert the expression form of the person parts extracted from the video content into an expression form with the determined level of detail, and reconstruct the video content.

2. The content construction method of claim 1, wherein the tables include a first table setting multiple levels of detail for the representation of images and a second table setting multiple levels of detail for the representation of audio, and the processor is caused to convert the representation of a person image corresponding to the person portion extracted from the video content into an representation of an image with a level of detail determined based on the first table, and convert the representation of audio corresponding to the person portion extracted from the video content into an representation of audio with a level of detail determined based on the second table, thereby executing a process to reconstruct the video content.

3. A content configuration method as described in claim 2, wherein the multiple levels of detail of the image expression format set in the first table include "live action," "3DCG," "illustration," and "schematic drawing," which have decreasing levels of detail, and the multiple levels of detail of the audio expression format set in the second table include "surround sound," "stereo," and "monaural," which have decreasing levels of detail.

4. A content composition device having a processor that executes the following processes: outputting video content to a user's output device; acquiring the user's operation time associated with the output of the video content; determining a level of detail corresponding to the acquired user's operation time based on a table in which multiple levels of detail are set so that the level of detail for expressing the video content decreases depending on the operation time; converting the expression form of a person image extracted from the video content into an expression form with the determined level of detail, and reconstructing the video content.

Images