Automatic video effect device, automatic video effect method, and image recording medium used therefor

Abstract

To provide an automatic video effect device and an automatic video effect method for automatically and autonomously performing rational and seamless video fusion by an advanced video effect using a predetermined information processing device in a next-generation video technology represented by AR (Augmented Reality) and MR (Mixed Reality).SOLUTION: An automatic video effect device includes a functional portion that decodes a video content and scenario of a parent video content that is a fusion source, functional portions that generate various pieces of data related to users and respective parent video objects generated from the decoding result, and a functional portion that performs predetermined video fusion effects such as selection of a parent video scene to be fused so as to satisfy a predetermined video fusion condition, determination of a child video object fusion position, video processing for video fusion, and construction of editing procedures on the basis of various pieces of data and casting data related to the child video object to be fused (data that clearly indicates the attributes, characteristics, etc. of the video object).SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an automatic video production device and an automatic video production method for processing and editing video content. [Background technology]

[0002] In recent years, "augmented reality ( A ugmented R "Augmented reality (AR)" and "mixed reality ( M updated R New next-generation images such as "AR (Augmented Reality; abbreviated as MR)" are attracting attention. In the field of these next-generation images, for example, in the aforementioned AR and MR, advanced imaging technology is required that can seamlessly fuse real space images (images actually seen by the user) with virtual images (predetermined image objects, etc.) in a real-time and rational manner.

[0003] On the other hand, with the advent of high-performance information processing devices, such as artificial intelligence (AI), it is becoming a reality for information processing devices to automatically or autonomously perform image processing tasks that have traditionally been performed by humans, such as image processing and editing, and several technologies for this have been disclosed.

[0004] Background art in this technical field is, for example, Patent Document 1. Patent Document 1 discloses a video system technology for composing moving image sequences, in which a predetermined information processing device such as a computer issues a series of shooting instructions to a shooting device according to a predetermined production procedure, and automatically edits the shot moving image according to the group of instructions. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2010-232813 A Summary of the Invention [Problem to be solved by the invention]

[0006] Patent Document 1 discloses a technology for a predetermined video processing device to process and edit video in accordance with a predetermined production scheme (a framework showing the contents and procedures of the production). However, there is no disclosure of a video technology for automatically and autonomously performing a video fusion production that can achieve a predetermined appealing effect for viewers and users by embedding and integrating video objects without compromising the story and content image of the original video content.

[0007] In particular, as in the cases of AR and MR mentioned above, one of the technical challenges in the field of next-generation video is the creation of advanced video fusion means that can seamlessly and rationally fuse a specified virtual image into a scenario-indefinite real-world image, using the real-world image as the source for embedding or fusion of the image.

[0008] The present invention has been made to solve the above-mentioned problems, and its object is to provide an automatic video production device and an automatic video production method for automatically or autonomously executing the above-mentioned advanced video fusion production, and a video recording medium for use therein. [Means for solving the problem]

[0009] In view of the above-mentioned background technology and problems, the present invention provides, as an example, an automatic video production device that performs an embedding or fusion process of a specified child video object not included in a parent video content into a specified video scene of the parent video content that is the source of the video fusion, and is equipped with a video content decoding unit that decodes the video content of all or part of the parent video content, or divides the parent video content into specified parent video scenes, and generates specified parent video decoding data that digitizes the results of the parent video content decoding or the parent video scene division results in a specified description format. Effect of the Invention

[0010] According to the present invention, advanced image production, processing, and editing can be automatically and autonomously executed to realize rational and seamless image fusion. [Brief description of the drawings]

[0011] [Figure 1] 1 is a configuration block diagram of an automatic video production device according to a first embodiment. [Diagram 2] 1 is a processing flowchart of a cast data and script data generation process in the first embodiment. [Diagram 3] 2 is a schematic diagram showing the contents and data structure of cast data in Example 1. FIG. [Figure 4] 2 is a schematic diagram showing the contents and data structure of script data in the first embodiment. FIG. [Diagram 5] 1 is a schematic diagram showing a data stream structure of video content into which cast data and script data have been embedded in Example 1. FIG. [Figure 6] 1 is a processing flowchart of a video fusion production, processing, and editing process in the first embodiment. [Figure 7] 1 is a schematic diagram showing an outline of a specific implementation example of the video fusion production, processing, and editing process in the first embodiment, broken down by processing steps. [Figure 8] 13 is a flowchart of a processing process in the second embodiment. [Figure 9] FIG. 11 is a schematic diagram conceptually showing a flow of a treatment process in Example 2. [Figure 10] FIG. 11 is a configuration block diagram of an automatic image production device according to a third embodiment. [Figure 11] 13 is a flowchart of a processing process in the third embodiment. [Figure 12] FIG. 13 is a configuration block diagram of an automatic image production device according to a fourth embodiment. [Figure 13] 13 is a flowchart of a processing process in the fourth embodiment. [Figure 14] FIG. 13 is a schematic diagram showing an example of AR display using a glasses-type HMD in the fourth embodiment. [Figure 15] FIG. 13 is a configuration block diagram of an automatic video production device according to a fifth embodiment. [Figure 16]13 is a flowchart of a processing process in the fifth embodiment. [Figure 17] FIG. 13 is a conceptual diagram of a CM video fusion service system using an automatic video production device in a sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. EXAMPLES

[0013] Fig. 1 is a block diagram of the automatic video production device in this embodiment. In Fig. 1, the automatic video production device 80 surrounded by a dashed line is composed of various functional parts such as a video content data reading unit 1, a video content decoding unit 2, a video object identification and extraction unit 3, a viewer biometric data detection unit 4, a viewer gesture and behavior data detection unit 5, a surrounding environment data detection unit 6, a cast data generation unit 7, a script data generation unit 8, a child (fusion target) video object data storage unit 9, a processing data storage unit 10, a video fusion production unit 11, a video processing, editing and rendering unit 12, and a control unit 20. In addition, each functional part is connected to a common data bus 25 as shown in the figure, and data signals and control signals can be exchanged between predetermined functional parts via this data bus 25.

[0014] Next, specific functions of each functional unit will be described. In this embodiment, the video content that is the source of the video fusion (hereinafter, for the sake of simplicity, referred to as parent video) is a produced video content such as a drama or a recorded live video of an artist.

[0015] The video content data reading section 1 in the automatic video production device 80 has a function of reading the video data of the parent video content 50.

[0016] The video content decoding unit 2 also has the function of decoding the video content and scenario from the video data of the parent video content 50 obtained from the video content data reading unit 1, and further dividing the entire parent video content into specified video scenes based on the decoding results.

[0017] The video object identification and extraction unit 3 also has the function of identifying and extracting some or all of the video objects, including people, objects, backgrounds, and scenes appearing in the parent video content 50, as individual video objects, and assigning each extracted video object a specified object identifier, such as an object name or ID number, to distinguish it from other video objects.

[0018] Meanwhile, the viewer biological data detection unit 4 has a function of detecting and acquiring biological data of the viewer or user 60 such as respiratory rate, heart rate, body temperature, blood pressure, and amount of sweating by a predetermined biological sensor 15 .

[0019] The viewer gesture and behavior data detection unit 5 also has the function of detecting and acquiring various data relating to the gestures or behavior of the viewer or user 60, such as the viewer's gaze direction and changes in facial expression, using a specified visual sensor 16, such as a viewer monitor camera.

[0020] Furthermore, the surrounding environment data detection unit 6 has a function of detecting and acquiring data related to the state of the surrounding environment of the viewer or user 60, such as temperature, humidity, air pressure, and illuminance (brightness), using a predetermined environmental sensor 17. Note that these detected and acquired data are not limited to the data described above, and any data item may be added or selected as necessary. Naturally, it is not necessary for all of the detection units to be equipped in the automatic video production device 80, and they can be added or selected as necessary according to the required data items, and as will be described later, if none of these detected and acquired data are required, it is of course not necessary to equip all of them.

[0021] Next, the cast data generation unit 7 generates data (hereinafter referred to as cast data for simplicity) that explicitly indicates the type, features, character, etc. of each video object (hereinafter referred to as parent video object for simplicity) in the parent video identified and extracted by the video object identification and extraction unit 3, as well as the video object (hereinafter referred to as child video object) that is to be merged with the parent video, and has the function of linking each of these to the corresponding video object.

[0022] Similarly, the script data generating unit 8 has a function of generating data (hereinafter referred to as script data for simplicity) for each video object appearing in each video scene of the parent video divided by the video content decoding unit 2, which data is analyzed based on the broadcast contents and scenario of the parent video also decoded by the video content decoding unit 2 and specifies the position, movement, and so on of the video object in accordance with a predetermined timetable, and linking the data to the corresponding video scene and video object. The specific contents of this script data and the above-mentioned cast data will be described later.

[0023] Next, the child image object data storage unit 9 and the process data storage unit 10 are each constituted by a predetermined storage device or storage device such as a server, or a predetermined data storage medium such as a magnetic disk or optical disk.

[0024] The child video object data storage unit 9 stores and memorizes the video data of the child video objects. The child video object stored here may be a single video object, or multiple video objects may be stored in the form of a library so that the automatic video production device 80 can freely select them depending on the situation of the viewer and his / her surrounding environment, or the broadcast contents and scenario of the parent video content. Furthermore, these stored child video objects may be changed or updated successively depending on predetermined conditions such as the passage of time or the change of seasons.

[0025] On the other hand, the processed data storage unit 10 appropriately stores various data generated or processed by the automatic image production device 80, such as cast data and script data, or a data stream of a parent image incorporating each of these data.

[0026] For the sake of convenience in FIG. 1, the child video object data storage section 9 and the processed data storage section 10 are shown as separate storage devices, but it goes without saying that they may be configured to share the same storage device.

[0027] 1, the child video object data storage unit 9 and the processing data storage unit 10 are configured to be built into the automatic video production device 80, but the present embodiment is not limited to such a configuration. For example, it is perfectly acceptable to configure these data storage units to be installed in an external server such as a cloud, and to exchange data with and control the automatic video production device 80 via a predetermined communication means or network.

[0028] Next, the video fusion production unit 11 has a function of executing a "video fusion production" process for integrating a child video object into a parent video content using various data such as cast data, script data, etc. The specific contents of this "video fusion production" process will be described later.

[0029] Next, the video processing, editing and rendering unit 12 has a function of actually executing processing such as video processing, editing or rendering (video drawing) based on the result of the "video fusion production" processing. Note that this video processing, editing or rendering processing can be executed by making full use of predetermined digital image processing technologies such as computer graphics technology, 3D morphing technology, shading technology, etc., so detailed explanations thereof will be omitted here.

[0030] Finally, the control unit 20 is connected to each of the functional units via a data bus 25 and has a function of appropriately controlling the functions.

[0031] As explained above, in the embodiment of Fig. 1, an example in which each functional unit is divided into blocks is shown for the convenience of explanation, but it goes without saying that the automatic image production device 80 of this embodiment does not actually need to be configured with divided functional blocks as shown in the figure. For example, it is perfectly acceptable to configure the processing in each functional unit to be comprehensively executed using a specified information processing device such as AI.

[0032] Furthermore, when an information processing device such as an AI is used as the automatic image production device 80 of this embodiment, by making full use of the latest AI technology such as deep learning technology, it is possible to carry out each of the above processing processes with advanced estimation and judgment capabilities that are closer to human thinking.

[0033] Finally, the parent video content in which the child video objects have been fused through the above series of processes is played back by a video playback unit 13 and is played back, distributed, or broadcast to a viewer or user 60 via a specified display device 14.

[0034] Next, an embodiment of the processing of the automatic image rendering device 80 in this embodiment will be described.

[0035] The series of processing steps in this embodiment are divided into a two-stage processing process: first, a "casting data, script data generation process" in which casting data and script data are generated for the entire parent video content; and then, a "video fusion production, processing, and editing process" in which the casting data and script data generated in this processing process are used to perform video fusion production and video processing, editing, and rendering processing, and further, the parent video content with child video objects fused therein as necessary is played back, distributed, or broadcast to viewers or users 60.

[0036] First, the process flow of the "cast data and script data generation process" will be described. Fig. 2 is a flowchart of the cast data and script data generation process in this embodiment.

[0037] 2, first, in processing step 101 (S101, hereinafter processing step will be abbreviated as S), the entire parent video content and all video data of the child video objects are read.

[0038] Then, in the next step S102, some or all of the video objects appearing in the parent video content 50, including people, objects, backgrounds, and scenery, are identified and extracted as individual parent video objects, and each extracted parent video object and each separately loaded child video object are assigned a specified object identifier such as an object name or ID number to distinguish them from other video objects.

[0039] In the next step S103, the video contents and scenario of the entire parent video content, i.e., the story, world view, or video production framework of the content, are decoded from the position, movement, and dialogue of each parent video object identified and extracted in S102, and the entire content is appropriately divided into predetermined video scenes based on the result of the decoding. The decoded video content information, scenario information, or video scene division information is then converted into data in a predetermined data description language or format and appropriately stored in the processed data storage unit 10.

[0040] Then, in the next step S104, predetermined cast data, as described below, is generated for each parent video object (hereinafter referred to as parent video object) and child video object identified and extracted in the previous processing step.Furthermore, in the next step S105, the parent video cast data is linked to the corresponding parent video object and incorporated into the video data stream of the parent video content.Furthermore, the cast data for the child video objects is also linked to the corresponding parent video object and written to the child video object data storage unit 9, the processed data storage unit 10, or a predetermined data storage unit other than the above.

[0041] After that, in the next step S106, the scene number N is set to 1, and in the next step S107, the Nth scene from the beginning of the parent video content is selected.

[0042] Furthermore, in the next step S108, specific script data as described below is generated for each parent video object that appears in the video scene of the parent video, and in the next step S109, the script data is linked to the corresponding video scene and video object and then incorporated into the video data stream of the parent video content.

[0043] Here, an embodiment regarding the contents and configuration of the cast data and the script data will be described.

[0044] FIG. 3 is a schematic diagram showing the contents and data structure of the cast data generated in S104 in this embodiment.

[0045] Casting data is data generated to enable the automatic video production device 80 to reliably recognize the position and role that each parent video object and child video object identified and extracted in S103 plays within the parent video content or on its own.

[0046] Therefore, the cast data is generated for each video object based on the video content information or scenario information of the parent video content decoded in S103 or the analysis information of the video object itself, and includes, for example, items such as those shown in FIG. (a) Physical Attributes (items that specify the physical classification or attributes of the object) (b) Social Attributes (for people and animals, social attributes, personality, character, and other items that characterize the person or animal in the video content, or items related to objects that cannot be expressed by physical attributes) (c) Visible state (items that clearly indicate the state that can be recognized from the image, such as shape, size, color, etc.) (d) Invisible conditions (items that indicate conditions that are impossible or difficult to recognize through images, such as weight, temperature, and texture) (e) correlation parameters (items that specify relationships between objects, such as interpersonal or social relationships, or physical or chemical interactions between objects) The necessary information regarding the classification, attributes, role, positioning, characteristics, character, etc. of each video object in the video content is digitized or parameterized, and is composed of a group of data or parameters that are linked to and comprehensively managed for each corresponding video object.

[0047] Meanwhile, FIG. 4 is a schematic diagram showing the contents and data structure of the script data generated in S108 in this embodiment.

[0048] The script data is data generated to allow the automatic video production device 80 to reliably recognize information about each parent video content that is equivalent to a script created and used in a drama or a play. Therefore, the script data is generated for each video scene and each video object that appears in the scene based on the video content information and scenario information for each scene of the parent video content decoded in S103 or the analysis information of the video object itself, and includes items such as those shown in FIG. (a) The three-dimensional position of the video object in the scene. (b) The orientation and posture of the video object (c) The lines of the video object (if the object has lines), facial expressions, gestures, and actions (d) Predetermined constraints imposed on each video object or on the entire scene. Information regarding the position and movement of each video object within the video scene, or the constraints imposed on them, is converted into data or parameters, and is composed of a group of data or parameters that are linked to and managed comprehensively for each corresponding video scene and video object.

[0049] The above-mentioned cast data and script data items are merely examples, and are of course not limited to these. Depending on the needs of the video production described below, information on a given item may be converted into data or parameters and added, or may be selected arbitrarily.

[0050] Furthermore, with regard to the data description language or format used to describe this casting data, script data, or the video content information data, scenario information data, or video scene division information data, etc., any description language or format may be used as long as it can be recognized by the automatic video production device.

[0051] Here, we return to the explanation of the processing flow of the "casting data, script data generation process" in Figure 2. Following S109, in S110 it is determined whether or not the generation and incorporation of script data for all scenes of the parent video is complete. If the determination is "Yes", in the next S111 the parent video data stream into which the cast data and script data have been incorporated is written to the processed data storage unit 10 or another specified data storage unit, and further the cast data linked to each child video object is also written to the child video object data storage unit 9 or the processed data storage unit 10. On the other hand, if the determination is "No", in S112 the process N=N+1 is performed, and the process returns to S107 to repeat the series of processing steps.

[0052] Fig. 5 shows the data stream structure of the parent video content in this embodiment in which the cast data and script data are incorporated. Fig. 5(a) shows an example of a data structure in which the cast data and script data are all placed at the beginning of the data stream, and Fig. 5(b) shows an example of a data structure in which only the cast data is placed at the beginning of the data stream, and the script data is placed at the beginning of each video data segmented for each scene.

[0053] 5 shows only two examples, and the data stream structure is not limited to the examples. Naturally, any data structure can be used as long as the automatic image production device 80 can freely read and write.

[0054] Finally, when the generation and incorporation of cast data and script data for all scenes of the parent video content is completed according to the above processing flow, a determination is made in S113 as to whether or not to end the series of "cast data, script data generation processes." If the determination is "Yes," the series of "cast data, script data generation processes" ends. On the other hand, if the determination is "No," the process returns to S101, and the series of "cast data, script data generation processes" is then repeated for another parent video content.

[0055] Next, the process flow of the other process, "video fusion production, processing, and editing process," will be described. Fig. 6 is a flowchart of the video fusion production, processing, and editing process in this embodiment.

[0056] In Figure 6, first, in S121, the video data stream of the parent video content with the embedded casting data and script data generated in the above-mentioned "casting data, script data generation process" and stored in a specified data storage unit, and all child video object data (video data of the child video objects and casting data linked to the child video objects) are read.

[0057] The parent video data stream with the cast data and script data incorporated therein is not limited to the video data stream generated in the above-mentioned "cast data, script data generation process." Naturally, it may be generated by other video equipment or processing processes, and for example, cast data and script data created manually and the original video content data stream may be read separately.

[0058] In the next step S122, biometric data such as the respiration rate, heart rate, body temperature, blood pressure, and amount of sweat of the viewer or user 60 are detected and acquired from the viewer biometric data detection unit 4. At the same time, data on the gestures or behavior of the viewer or user 60, such as the line of sight and facial expression, are detected and acquired from the viewer gesture and behavior data detection unit 5, and various data on the environmental conditions surrounding the viewer or user 60, such as temperature, humidity, air pressure, and illuminance (brightness), are detected and acquired from the ambient environment data detection unit 6.

[0059] The types of detection data are not limited to those in this embodiment, and may be added or selected as necessary.

[0060] Next, in S123, the automatic image production device 80 itself estimates the health condition, physiological condition, and psychological state of the viewer or user 60, such as emotions, preferences, desires, and objects of interest, from the data detected and acquired in S122.

[0061] Of course, the automatic image production device 80 in this embodiment is not limited to the above-described estimation results of the health, physiological, and psychological states of the viewer or user 60 as the selection conditions for the optimal image object, and any selection conditions may be used for selection. In this case, the data to be acquired in S122 is naturally not limited to the above-described data, and may be changed arbitrarily according to the selection conditions, and depending on the selection conditions, such data on the viewer's situation may not be required. In such a case, it is of course possible to skip S122 or S123 or omit the processing steps themselves.

[0062] Then, in the next step S124, the automatic image production device 80 itself determines what child image object should be embedded or integrated into the parent image at that time in order to produce the most appropriate appealing effect, based on the detected and acquired data, or the health, physiological, and psychological state of the viewer or user 60 estimated therefrom, and selects the most appropriate child image object from among one or more child image objects.

[0063] Furthermore, in the next step S125, a source video scene into which the selected child video object is to be merged is selected from among the scenes of the parent video content.

[0064] In the next step S126, the screen position in the video scene where the target child video object is to be embedded or blended is designated.

[0065] Particular care must be taken when a parent video object exists at the position in the video scene where the child video object is to be embedded or blended. In such a case, it is necessary to carry out a specific video processing or editing process, paying attention to the anteroposterior and superimposition relationships between the child video object and the parent video object, so that the two video objects appear to be embedded or blended naturally and seamlessly without any sense of incongruity. The next step, S127, determines the framework for the video production through this video processing or editing process.

[0066] In other words, in S127, a "video processing editing scheme" is constructed which clearly specifies the specific video processing and editing content and procedures to be applied to the child video object or parent video object or other parent video screen in order to blend the selected child video object naturally and seamlessly into the target parent video object or screen position without any awkwardness.

[0067] The four processing steps S124, S125, S126, and S127 are the processing steps executed mainly by the image fusion production unit 11 in the block diagram of this embodiment shown in Fig. 1, and are the most important constituent elements of this embodiment. For the sake of convenience, these processing steps will be collectively referred to as "image fusion production" steps below.

[0068] In this "video fusion production" process, based on the casting data and script data for each parent video object loaded in S127, the casting data for each child video object, and also the various data obtained in S122 and the estimated health, physiological and psychological state of the viewer or user 60 obtained in S123 based on that data, the automatic video production device 80, at its own discretion, selects the child video objects and scenes to be fused with the parent video, specifies the positions at which the child video objects are fused, and constructs a "video processing editing scheme."

[0069] When each process is executed, an optimum process is performed so as to clear the "image fusion conditions" as shown below, for example. (a) It must not interfere with the storyline or image of the content evoked by the broadcast content or scenario of the parent video content. (b) The fusion of child video objects produces a desired appealing effect. (c) The fusion of child image objects does not result in irrationality or physical contradictions. (d) The fusion of child video objects does not cause viewers to feel uncomfortable or disgusted. In addition, in the above examples of "image fusion conditions," "the absence of irrationality or physical contradictions" in (c) means, specifically, that there are no irrational images, such as an object floating in the air without any support, or a physical object passing through a wall.

[0070] However, if it is deemed appropriate to deliberately use fantasy or science fiction visual effects in order to clear other "image fusion conditions," then even irrational or physically contradictory effects such as those mentioned above will be permitted.

[0071] In this way, the "image fusion conditions" can be loosened or strengthened as appropriate, taking into consideration the balance of each condition. Furthermore, the above items (a) to (d) of the "image fusion conditions" are merely examples, and the conditions are not limited to these items. The user or manager of the automatic image production device 80 may add, delete, or change the conditions at will.

[0072] Next, in S128, following the "video fusion production" step, it is determined whether or not to actually execute video fusion processing such as video processing, editing, and rendering based on the processing result. If "Yes", video processing, editing, or rendering is actually executed in the next S129. As described above, this video processing, editing, or rendering can be executed by making full use of a predetermined digital image processing technology such as computer graphics technology, 3D morphing technology, and shading technology, so a detailed description thereof will be omitted. On the other hand, if "No", this S129 is skipped, and in the next S130, it is determined whether or not a series of "video fusion production, processing, and editing processes" has been completed for the entire parent video content. If the determination is "Yes", the processing flow returns to S122 again and the series of processes is repeated. On the other hand, if "No", the flow proceeds to the next S131.

[0073] In S131, the selection result of the parent video scene to be fused obtained in the "video fusion production" step, the designation result of the parent video object to be fused or the screen position, and further the "video processing and editing scheme" and other data are written as predetermined video fusion production data in the processed data storage unit 10. At the same time, the video data stream of the parent video content for which a series of video fusion processes have been completed is also written to the processed data storage unit 10 or another predetermined data storage unit.

[0074] In the next step S132, it is determined whether or not to play the parent video content for which a series of video fusion processes have been completed. If the determination is "Yes," the video is actually played, distributed, or broadcast in the next step S133. On the other hand, if the determination is "No," this step S133 is skipped, and the series of processing flows related to the "video fusion production, processing, and editing process" is completed.

[0075] In this embodiment, for the sake of convenience, a process flow is described in which the parent video content that has been subjected to the video fusion processing is played back or distributed or aired after the series of "video fusion production, processing, and editing processes" is completed for the entire parent video content, but this embodiment is not limited to this. In other words, it is of course possible to use a system in which the parent video content that has been subjected to the video fusion processing is played back or aired in parallel with the "video fusion production, processing, and editing processes" being performed first.

[0076] Next, a specific processing procedure for each processing step of the "image fusion production, processing, and editing process" explained in FIG. 6 will be explained using a specific processing example.

[0077] Fig. 7 is a schematic diagram showing the specific processing procedure for each processing step in the "Video Fusion Production, Processing, and Editing Process" for a certain video fusion processing example. The processing example introduced here is a case where a live concert video of a certain idol (pseudonym (A)) is taken as the produced parent video content that is the source of the video fusion, and a commercial video for a product sold by a certain beverage manufacturer (pseudonym (B)) is incorporated into the produced live concert video and distributed and broadcast. Note that this produced parent video content has already been generated with the specified cast data and script data by the above-mentioned "cast data, script data generation process", and incorporated into the video data stream.

[0078] In Fig. 7, the characters in the processing step column correspond to the names of the processing steps explained in Fig. 6, and the processing content column gives an overview of the processing content in the processing step. And the specific processing example column explains the specific processing content in this example using text and schematic diagrams.

[0079] In FIG. 7, when a request for CM video presentation is received from company (B), the live concert video data stream in question, in which cast and script data have been incorporated, is first read into automatic video production device 80 (S121).

[0080] Next, the automatic image production device 80 detects and acquires data on the viewer's biological condition, facial expression, gaze target (= object of interest), surrounding environment, etc. from various sensors (S122).

[0081] From the detected and acquired data, the automatic image production device 80 estimates that the viewer 60 is currently thirsty and wants some kind of drink, and determines that incorporating a video of drinking water into the live video will have a high commercial appeal (S123).

[0082] Therefore, a video of a soft drink plastic bottle (C) made by company (B) is selected as a child video object (commercial video object) to be incorporated into the parent video, and the video object data is read (S124).

[0083] Next, the automatic video production device 80 recognizes that the parent video content is a video of a live concert starring an idol (A), and decides to naturally blend a video of a child video object, a plastic bottle (C), into a video scene of the idol (A) dancing during the live concert, as a video production that does not impair the image of the content and achieves a desired commercial appeal effect.A dance scene that will be the source of the video fusion is selected from the video scenes of the parent video content (S125).

[0084] Furthermore, within the selected dance scene, the right hand of the idol (A) is specified as the position for fitting a specific child video object (plastic bottle (C)). In order to achieve a natural and seamless video fusion, it is decided to change the direction to a video in which the idol (A) dances while holding the plastic bottle (C) in his right hand (S126).

[0085] The automatic image production device 80 then, at its own discretion, constructs a "image processing and editing scheme" that specifies the specific content and steps of what image processing or editing should be applied to the parent image and child image object in order to change the performance from the original parent image, which is holding nothing, to an image of the child dancing with a plastic bottle (C) in his right hand (S127).

[0086] Here, the outline of the "video processing and editing scheme" is as shown in Figure 7, for example. (a) (A) The overlapping state (front-back relationship) of each finger of the right hand and the plastic bottle is aligned. (b) The image of the open right hand in (A) is processed into an image of the hand holding a plastic bottle. (c) Adjusting the illumination and glare of the light shining on the plastic bottle (d) Adding a shadow cast by a plastic bottle and modifying the shadow cast by the fingers The above is just one example and overview of a "video processing and editing scheme," and in reality, a more specific and detailed scheme will be constructed.

[0087] The "video processing editing scheme" constructed in this way is then applied to actual video footage by making full use of predetermined digital image processing techniques such as computer graphics technology, 3D morphing technology, and shading technology (S129).

[0088] Finally, the parent video content in which the child video object (plastic bottle (C)) has been correctly fused, i.e., live video including a scene of the idol (A) dancing while holding the plastic bottle (C) in a specific dance scene, is distributed and broadcast to the viewers 60 (S133).

[0089] As described above, by using the automatic video production means or device of this embodiment, it is possible to naturally and seamlessly integrate a specific video object that is independent of the video content into the video content without interfering with the story or content image of the video content.

[0090] Furthermore, according to the automatic video production device of this embodiment, it is possible to flexibly respond by selecting the video objects to be blended to suit the emotions, preferences, desires, interests, etc. of each individual viewer or user, or by appropriately executing video fusion that is effective at each moment in accordance with changes in the viewer's or user's situation in almost real time (for example, while the viewer or user is currently viewing the video content in question).

[0091] Considering these characteristics, therefore, it can be said that the automatic video production device of this embodiment is extremely effective for incorporating commercial videos into video content such as that introduced in FIG.

[0092] When the automatic video production device of this embodiment is used for embedding commercial videos into predetermined video content, the number of times or the time that the target commercial video is embedded or integrated into the parent video content can be counted, and depending on the counting result, restrictions can be placed on commercial provision, or a mechanism can be combined to charge or bill a predetermined commercial fee to the commercial sponsor. Of course, the automatic video production device of this embodiment is not limited to the above-mentioned use of embedding commercial videos into video content, and can be used for any purpose. EXAMPLES

[0093] In Example 1, a two-stage embodiment is implemented in which first, a "casting data, script data generation process" is used to generate casting data and script data for the entire parent video content, and then this is incorporated into the parent video data stream. After that, the "video fusion production, processing, and editing process" is used to perform video fusion processing of the child video objects.

[0094] On the other hand, it is also possible to pre-read the parent video content and embed the child video object therein while playing, distributing or broadcasting the parent video content.

[0095] Therefore, in this embodiment, an example is described in which an automatic video production device that simultaneously performs playback or distribution, broadcast processing of parent video content and video fusion production, processing, and editing processing of the parent video content is installed in a video content supply server that broadcasts or distributes the parent video content.

[0096] The configuration block diagram of the automatic image production device in this embodiment is the same as that in FIG. 1, so it is omitted.

[0097] FIG. 8 is a flow chart of the processing steps in this embodiment.

[0098] In this embodiment, as a preliminary step before the start of a series of processing steps, for each child video object stored in the child video object data storage unit 9, predetermined data or a group of data that have been digitized or parameterized with predetermined information regarding the classification, attributes, characteristics, etc. of the child video object, i.e., data equivalent to the casting data described in the embodiment of the ``Cast Data, Script Data Generation Process'' in Figure 2, are stored in a predetermined data storage unit in a form linked to the child video object.

[0099] In FIG. 8, first, in S201, the automatic video production device 80 reads all child video object data, that is, the video data of the child video objects and the cast data linked to the child video objects, from a predetermined data storage unit.

[0100] Next, in S202, the reading of the video data of the parent video content is started. Then, in the next S203, the automatic video production device 80 starts playing the parent video content, and simultaneously reads in advance the video data of the parent video to be played, distributed, or broadcast within a predetermined time, for example, several seconds to several tens of seconds, from the currently played scene.

[0101] Then, in the next step S204, similar to S102 described in Figure 2, all objects, including people and objects that appear newly in the previously loaded image, as well as the background and scenery, are identified and extracted as individual video objects, and each extracted video object is assigned a specified object identifier, such as an object name or ID number, to distinguish it from other video objects.

[0102] Next, in S205, the video content and scenario of the pre-loaded video are decoded from information data such as the position, movement, and dialogue of each parent video object identified and extracted in S204, and the video content and scenario are appropriately divided into specified video scenes as necessary based on the decoded information.

[0103] S205 is mainly executed by a functional unit corresponding to the video content decryption unit 2 in the block unit of the automatic video production device 80 shown in Fig. 1. In the first embodiment, the video content decryption unit 2 has a function of once reading all of the parent video content and then decoding the broadcast contents and scenario and performing scene division processing for the entire video content. However, the video content decryption unit 2 in this embodiment can be realized by a function of decoding the video contents and scenario and performing scene division processing using the video data that has been pre-read from the start of the video playback or distribution or broadcast of the parent content to that point, that is, the video data up to the middle stage of the video content, as described above. In this respect, the functions of this embodiment and embodiment 1 are slightly different.

[0104] In the case of video decoding in the middle of such content, naturally, video content information and scene division information are added as the reading and decoding of the parent video data progresses. Based on this additional information, the scenario information of the video content from the start of distribution or broadcasting to that point is updated sequentially by making predetermined additions, changes or corrections.

[0105] Therefore, the video content decoding unit 2 in this embodiment or the control unit 20 that controls it also has the function of sequentially updating such video content information, scenario information, scene division information, etc. to the latest information data to which additions, changes, or modifications have been made as described above, and then storing it in a specified processing data storage unit 10.

[0106] In the next step S206, it is determined whether the preloaded parent video scene is the parent video scene into which the child video object should be embedded or merged. If the determination is "Yes," the process proceeds to S207 (described later) and thereafter. On the other hand, if the determination is "No," steps S207 to S210 (described later) are skipped and the process jumps to S211.

[0107] In the next step S207, if there are multiple types of child video objects, a child video object to be embedded or merged into the selected target parent video scene is selected from among them.

[0108] Then, in the next step S208, a screen position in the fusion-target parent video scene into which the target child video object is to be embedded or merged is designated.

[0109] Furthermore, in the next step S209, a "video processing editing scheme" is constructed which specifies the specific content and procedures of video processing and editing to be applied to the child video object or parent video object or other parent video screen in order to blend the selected child video object naturally and seamlessly into the target parent video object or screen position without any awkwardness.

[0110] The specific processing contents and procedures in each of these processing steps are exactly the same as the so-called "image fusion performance" processing steps explained in S124 to S127 of FIG. 6, so detailed explanations thereof will be omitted here.

[0111] However, in this embodiment, unlike the first embodiment, a series of processes are executed from the video data of the parent video content that has been pre-read in parallel while the parent video content is being played, distributed, or broadcast, so there are more time constraints than in the first embodiment and faster processing is required.

[0112] Therefore, in each judgment, selection and designation processing step in S206, S207 or S208, a method can be used in which, for example, part or all of the casting data on the child video object side is used as a predetermined keyword, and the suitability or appeal of each parent video scene to that keyword is evaluated using a predetermined numerical value or parameter, and the magnitude of this is used as the criterion for judgment or selection, thereby enabling high-speed processing.

[0113] Furthermore, in the "video processing and editing scheme" construction processing step in S209, the constructed "video processing and editing scheme" may be partially simplified or omitted.

[0114] For example, a "video processing and editing scheme" may be a video fusion presentation that specifies only the anteroposterior or overlapping relationship between a child video object to be inserted into a specified video scene of a parent video and an existing parent video object within that parent video scene, or a video fusion presentation that simply inserts a selected specified child video object or part of it into a specified screen position within a specified video scene of an also selected parent video, omitting all processing and editing steps for the video itself.

[0115] Next, based on the "video fusion performance" results constructed as above, the video is actually processed, edited, and rendered in S210. As explained in S129 in the implementation examples of Figures 6 and 7, these processes are performed by making full use of predetermined digital image processing technologies such as computer graphics technology, 3D morphing technology, and shading technology.

[0116] Then, in the next step S211, a process is performed in which the parent video data after the above child video embedding or fusion process is sequentially written into a predetermined processing data storage unit 10. Here, in addition to the above video data, a process may be performed in which the parent video scene or child video object to be fused obtained in the above "video fusion production" process, or the result of designating the screen position in the parent video scene to be fused for embedding or fusion of the target child video object, and further various data related to "video fusion production" such as a "video processing and editing scheme" may be written or updated.

[0117] In the next step S212, the parent video data that has been written to the processed data storage unit 10 in step S211 and has undergone the embedding or fusion processing is read out, and the video scene next to the currently reproduced video scene is reproduced, distributed, or broadcast.

[0118] Then, in the next step S213, it is determined whether or not all the scenes of the parent video content have been played back, distributed, or broadcast. If the determination is "No," the process returns to step S203 and the series of processing steps is repeated. On the other hand, if the determination is "Yes," the series of processing steps ends.

[0119] FIG. 9 is a schematic diagram conceptually showing the flow of processing in the advance reading type automatic image rendering device shown in this embodiment.

[0120] As shown in FIG. 9, when a series of processing steps starts, the automatic video production device 80 first reads the video data of the child video object to be embedded in the parent video content and its associated information, that is, the so-called cast data.

[0121] Next, the video data of the parent video content is pre-read for each predetermined video scene or video cut from the beginning, and for each parent video scene or cut that has been read, it is determined whether or not the target child video object should be fitted, and further the fitting position is scanned (searched). For example, this is the nth scene or cut.

[0122] In the next step, if the n-th parent video scene or cut is a target for fitting a child video object, a process is performed to actually fit the child video object into the designated position of the parent video scene or cut. At the same time, the above-mentioned look-ahead process and other processes are performed on the next n+1-th parent video scene or cut.

[0123] In the next step, while the nth parent video scene or cut is being played back, the child video object embedding process is simultaneously performed on the n+1th parent video scene or cut, and the pre-reading process and other processes are performed on the n+2th parent video scene.

[0124] A processing process in which multiple processing steps are performed sequentially and simultaneously in this manner is generally referred to as pipeline processing, and by adopting this type of pipeline processing in this embodiment, it is possible to play, broadcast, or distribute a specified parent video content while simultaneously embedding a specified child video object into the parent video content in almost real time.

[0125] As described above, the automatic video production device in this embodiment has a processing process that seamlessly carries out the ``casting data, script data generation process'' shown in Figure 2 in embodiment 1 and the ``video fusion production, processing, and editing process'' shown in Figure 6.

[0126] However, in this embodiment, the series of processes of generating cast data and incorporating it into the parent video data in S104 to S105 in Fig. 2, and generating script data and incorporating it into the parent video data in S108 to S109 are omitted. This is because an automatic video production device that performs embedding and fusion processing of child video objects into a pre-read parent video scene in parallel with the playback of a parent content as in this embodiment requires faster processing, and this is an example in which the series of processes are omitted as one of the measures to reduce the processing load. Therefore, if the processing capacity of the automatic video production device is sufficiently high, it is of course no problem to add the generation of cast data and script data and the embedding processing into the parent video data as in the embodiment of Fig. 6. By adding such a process, when some child video object is embedded again into the target parent video content and fusion processing is performed later, more complex and advanced video fusion production processing using cast data and script data can be performed as in the embodiment 1 described in Fig. 6.

[0127] Similarly, in this embodiment, unlike the processing process of the first embodiment described in Fig. 6, the processing steps (S122 and S123 in Fig. 6) of detecting and acquiring data on the viewer's biological condition and surrounding environment from various sensors and estimating the health condition, physiological condition, and even psychological state of the target viewer, and the series of processes of reflecting the estimation results in the selection process of the child video object to be embedded in the parent video content and the video fusion performance process are omitted. As described above, this is also an example of omission as one of the measures to reduce the processing load in order to achieve faster processing. Therefore, if the processing capacity of the automatic video performance means or device is sufficiently high, it is naturally possible to perform more advanced video fusion performance that detects and estimates viewer-related information and reflects the estimation results, as in the embodiment of Fig. 6.

[0128] Furthermore, when an automatic video production device such as that of this embodiment is used for purposes such as incorporating commercial videos into video content, as in embodiment 1, it is possible to count the number of times or the time that the target commercial video is embedded or integrated into the parent video content, and depending on the counting results, it is possible to combine a mechanism for restricting the provision of commercials or for charging and billing a specified commercial fee to the commercial sponsor.

[0129] As described above, according to this embodiment, while playing back, distributing or broadcasting a parent video content, the parent video content can be pre-read in parallel to embed a child video object and perform fusion processing. EXAMPLES

[0130] In the second embodiment, an automatic video production means or device that plays, distributes, or airs a parent video content while simultaneously embedding and integrating a child video object is installed on the video content supply server side that airs or airs the parent video content. However, it is of course possible to install an automatic video production means or device with similar functions on the so-called edge terminal side, such as a smartphone carried by a general user.

[0131] In this embodiment, therefore, an automatic image production device that has the same functions as in the second embodiment and is installed in an edge terminal such as a smartphone will be described.

[0132] Fig. 10 is a block diagram showing the schematic configuration of the automatic image production device in this embodiment. In Fig. 10, the same numbers are used for functional units having the same functions as those in Fig. 1.

[0133] In FIG. 10, an automatic video production device 80 of this embodiment is built into an edge terminal 65 such as a smartphone carried by a user 60. In this automatic video production device 80, similar to the automatic video production device of the embodiment 1 shown in FIG. 1, each functional unit such as a video content data reading unit 1, a video object identification and extraction unit 3, a viewer biometric data detection unit 4, a viewer gesture and behavior data detection unit 5, a surrounding environment data detection unit 6, a processing data storage unit 10, a video processing, editing and rendering unit 12, and a control unit 20 are provided. In addition, each functional unit of the viewer biometric data detection unit 4, the viewer gesture and behavior data detection unit 5, and the surrounding environment data detection unit 6 is connected to a predetermined biosensor 15, a predetermined visual sensor 16 such as a viewer monitor camera, an environment sensor 17, etc. These functional units and sensors have the same functions as the corresponding functional units shown in FIG. 1, so detailed explanations thereof will be omitted here.

[0134] On the other hand, in this embodiment, a pre-read cut video outline decoding unit 32 is provided in place of the video content decoding unit 2 in FIG.

[0135] In this embodiment, it is necessary to decode the video information pre-loaded sequentially in almost real time, instead of reading the entire parent video content and then decoding the detailed video content of each video scene constituting the content as in the embodiment 1. Moreover, when the automatic video production device 80 is incorporated in the edge terminal side such as a smartphone as in this embodiment, there is a limit to its processing capability, making it difficult to perform the advanced video decoding process as in the embodiment 1.

[0136] Therefore, in this embodiment, the pre-loaded cut video summary interpretation unit 32 is provided with the function of extracting video cuts, which are video units that further subdivide the video scenes, from the video information that is pre-loaded sequentially, and quickly deciphering the summary of each cut from the types and arrangements of video objects that appear within the video cuts.

[0137] Furthermore, in this embodiment, instead of the image fusion production unit 11 of embodiment 1, high-speed processing is achieved by equipping it with an image embedding production unit 31 whose processing function is simplified by specializing the image fusion production to only the embedding of child image objects into parent image cuts, as described below.

[0138] In this embodiment, the video data of child video objects to be embedded in the parent video is stored in a child video object data storage unit 9 provided in an external server. Furthermore, for each of these child video objects, the cast data explained in "Cast Data and Script Data Generation Process" in Fig. 2, i.e., predetermined data or data groups obtained by digitizing or parameterizing predetermined information relating to the classification, attributes, characteristics, etc. of the child video object, is stored in the child video object data storage unit 9 in a form linked to the child video object. These child video object related data are then read into the automatic video production device 80 in this embodiment as appropriate by a predetermined communication means.

[0139] The external server in which this child video object data storage unit 9 is located may be a dedicated server managed by a commercial sponsor or a specified video content distribution company, etc., or it may be a general-purpose server such as a cloud connected via the Internet, etc.

[0140] Fig. 11 is a flow chart of the processing process in this embodiment. In Fig. 11, first, in S301, the automatic video production device 80 reads from the externally disposed child video object data storage unit 9 video data of each target child video object and predetermined data or data groups in which predetermined information on the classification, attributes, features, etc. of the child video object is digitized or parameterized, that is, data equivalent to the cast data explained in the embodiment of the "cast data, script data generation process" in Fig. 2, in a form linked to the child video object.

[0141] Next, in S302 and S303, like S202 and S203, reading of the video data of the parent video content is started, and playback of the parent video content is started, and simultaneously, video data of the parent video to be played, distributed or broadcast from the currently played scene until a predetermined time is read in advance.

[0142] In the next step S304, the parent video that has already been loaded is subdivided into a specified number of video cuts as described above, and for each of the subdivided video cuts, all objects including newly appearing characters and objects as well as backgrounds and scenery are identified and extracted as individual video objects, similar to S102 in the "casting data, script data generation process" described in Figure 2 of Example 1.

[0143] Next, in S305, the outline of the target parent video cut is deciphered from the type and position within the video of each parent video object identified and extracted in S304.

[0144] Then, in S306, similar to S122 in FIG. 8, various biometric data of the viewer or user 60 are detected and acquired from the viewer biometric data detection unit 4, data relating to the gestures or behavior of the viewer or user 60 from the viewer gesture and behavior data detection unit 5, and various data relating to the surrounding environmental conditions of the viewer or user 60 from the surrounding environment data detection unit 6.

[0145] Furthermore, in the next step S307, similar to S123 shown in FIG. 8, the automatic image production device 80 itself estimates the health condition, physiological condition, and psychological state, such as emotions, preferences, desires, and objects of interest, of the viewer or user 60 from the data detected and acquired in S306.

[0146] Of course, the automatic image production device 80 in this embodiment does not necessarily require the detection data and estimation results of the health, physiological, and psychological states of the viewer or user 60 as described above, and the detection data and estimation targets may be changed as necessary, and the processing process may not particularly require the data on the viewer or user. In such a case, it is of course possible to skip S306 and S307, or to omit the processing steps themselves.

[0147] In the next step S308, it is determined whether the pre-loaded parent video cut is the parent video cut into which the child video object is to be fitted. If the determination here is "Yes," the process proceeds to S309 (described later) and thereafter. On the other hand, if the determination is "No," steps S309 to S312 (described later) are skipped and the process jumps to S313.

[0148] In the next step S309, if there are multiple types of child video objects, one of them is selected as the object to be embedded or merged into the selected target parent video scene.

[0149] Furthermore, in the next step S310, the screen position at which the target child video object is to be embedded within the parent video cut to be embedded is specified.

[0150] Incidentally, when determining the parent video cut into which the child video is to be embedded in S308, or when selecting the child video object in S309, or when specifying the position into which the child video object is to be embedded in S310, a method can be used in which, for example, part or all of the casting data on the child video object side described above is used as a keyword, and the suitability or appeal of the keyword is quantified based on an overview of the target parent video cut, or, if necessary, the viewer's biometric data, gestures, behavioral data, surrounding environmental data, and even the estimated results of the viewer's situation detected in S306, and the magnitude of this value is used as the criterion for judgment or search, thereby enabling high-speed processing.

[0151] Then, in the next step S311, in order to fit the selected child video object into the specified parent video cut screen position as naturally and seamlessly as possible, a "video processing editing scheme" is constructed which specifies the specific video processing and editing content and procedures to be applied to the child video object or parent video cut screen.

[0152] However, in this embodiment, as in the second embodiment, a series of processes is executed from the video data of the parent video content that has been pre-read in parallel while the parent video content is being played back, distributed, or broadcast, so there are more time constraints than in the first embodiment, and faster processing is required. Therefore, in the "video processing and editing scheme" construction processing step of S311, as in S209, the "video processing and editing scheme" to be constructed may be partially simplified or omitted. Note that the details of this process are the same as in S209, so a description thereof will be omitted.

[0153] Next, based on the "video fusion performance" result constructed as above, in S312, the video is actually processed, edited, and rendered. These processes are the same as S210 in Fig. 6, so the description thereof will be omitted.

[0154] Then, in the next step S313, the parent video data having undergone the above-mentioned child video embedding process is sequentially written into the predetermined processed data storage unit 10. Note that the details of this process are the same as those of S211, and therefore a detailed description thereof will be omitted. Subsequent steps S314 and S315 are the same as those of S212 and S213 in FIG. 6, and therefore a detailed description thereof will be omitted.

[0155] As described above, according to this embodiment, an automatic image production device with simplified processing capabilities that is installed in an edge terminal such as a smartphone can be provided. EXAMPLES

[0156] In all of the first to third embodiments, the parent video to be fused is a predetermined pre-produced video content, such as a drama or a recorded live video, etc. In other words, the video content is a video content in which the story or scenario of the content, and further the video content of all the video scenes in the content, are already specified.

[0157] However, for example, in AR (augmented reality) and MR (mixed reality), which are attracting attention as next-generation video, head-mounted displays ( H ead M ounted D In AR / MR display devices such as HMD (Head-Mount Display), imaging technology is essential to embed or fuse a specified virtual image into the real image viewed by the user (HMD wearer).

[0158] In the case of such AR or MR, the image that is the basis for embedding or merging images, i.e. the parent image, corresponds to the real image that the user, i.e. the HMD wearer, sees in real time (hereinafter, the real image seen by the user will be referred to as the real visible image), but the image scenes or cuts change from moment to moment depending on the object that the user is seeing and the changes in the external environment around it, or the user's own gestures and actions, such as changes in the line of sight. Therefore, the contents and scenario of the parent image are also completely indefinite.

[0159] This embodiment describes an automatic video production device that can seamlessly and naturally embed or blend a specified child video object, i.e., a video equivalent to a virtual video in AR / MR, into a parent video whose video content, scenario, and even the video scene itself are undefined or fluid and change in real time.

[0160] Fig. 12 is a block diagram of the automatic image production device in this embodiment. In Fig. 12, the same reference numbers are used for functional units that have the same functions as those in Figs. 1 and 10.

[0161] As in the third embodiment, in advance preparation, video data of a virtual video to be embedded or merged with a parent video is stored as a child video object in a child video object data storage unit 9 provided in an external server. Furthermore, for each of these child video objects, the casting data explained in the "casting data and script data generation process" in Fig. 2, that is, predetermined data or a group of data obtained by digitizing or parameterizing predetermined information relating to the classification, attributes, characteristics, etc. of the child video object, is also stored in the child video object data storage unit 9 in a form linked to the child video object.

[0162] These child video object related data are then appropriately read into the automatic video production device 80 of this embodiment by a predetermined communication means.

[0163] The external server in which this child video object data storage unit 9 is located may be a dedicated server managed by a commercial sponsor or a specified video content distribution company, etc., or it may be a general-purpose server such as a cloud connected via the Internet, etc.

[0164] Further, the automatic image production device 80 in this embodiment is connected to an HMD 44 worn on the head of a user 60 as an AR / MR image display device, as shown in FIG.

[0165] It should be noted that this HMD 44 is available in two types: a see-through type in which the real image seen by the user with his or her naked eyes through a transparent display is used as the real visual image, and a video see-through type in which both the real visual image and the virtual image embedded or fused with it are video images. This embodiment is applicable to both types of HMD.

[0166] The HMD 44 is also equipped with sensors that detect the gestures and actions of the user 60, such as the movement of the user's gaze, sensors that detect the biometric information of the user 60, and sensors that detect environmental information such as the temperature and humidity around the user. For convenience, these sensors are collectively indicated as sensor 18 in FIG. 12. Furthermore, not all of the above sensors are essential to this embodiment, and they may be arbitrarily selected as necessary, and of course sensors that detect information different from the above sensors may be added. Conversely, the HMD 44 may not be equipped with any of these sensors.

[0167] Furthermore, the HMD 44 is equipped with an outside world capturing camera 19 capable of capturing images of the outside world in at least the same visual field as the image actually viewed by the user, and is connected to an automatic image production device 80 as an input device for parent image data.

[0168] The connection between the HMD 44, the sensor 18, and the external environment shooting camera 19 and the automatic image production device 80 is not limited to a wired connection as shown in FIG. 12, and any connection form that can exchange specified data signals, such as a specified wireless communication connection, may be used.

[0169] In FIG. 12, real-time image data corresponding to the image actually viewed by the user captured by the external world capturing camera 19 is sent to the automatic image production device 80 via the user actual field of view image reading unit 41 as parent image data to be fused.

[0170] The video object identification and extraction unit 3 has a function of sequentially identifying and extracting video objects that newly appear in each video cut from the real-time video data as individual video objects. Note that the video object identification and extraction unit 3 has the same functions as the functional units with the same numbers shown in Figure 1 or Figure 10, so a detailed functional description will be omitted here.

[0171] In addition, the real-time video cut decoding unit 42 is a functional unit equivalent to the video content decoding unit 2 in Figure 1 and the pre-loaded cut video outline decoding unit 32 in Figure 10, and has the function of decoding the contents of the loaded parent video, i.e., the image actually seen by the user in real time, in specified video cut units and converting it into data.

[0172] In interpreting this real-time actually visually recognized image, the automatic image production device 80 itself considers and judges the interpretation and digitization based on the image information data of the actually visually recognized image, as well as the history of information data such as the interpretation results of past images obtained up to that point and the scenario. The real-time image cut interpretation unit 42 may be provided with a function of constructing a scenario up to the current image scene based on the interpretation results, and successively updating the existing scenario data as the latest scenario data.

[0173] Meanwhile, each piece of detection data detected by the sensor 18 is acquired by the automatic image production device 80 via the viewer biometric data detection unit 4, viewer gesture and behavior data detection unit 5, surrounding environment data detection unit 6, etc. as necessary. Note that each of these detection units has the same functions as the functional units with the same numbers in Figure 1 or Figure 10, so a detailed functional description will be omitted here.

[0174] The video content data and scenario data that are successively generated or updated, as well as the various sensor data detected and acquired by the sensor 18, are stored and memorized in the processed data storage unit 10 as appropriate.

[0175] 12, the processed data storage unit 10 is configured to be built inside the automatic video performance device 80, but this embodiment is not limited to this configuration, and for example, these data storage units may be installed in an external server such as a cloud, and data may be exchanged with and controlled by the automatic video performance device 80 via a predetermined communication means or network. Furthermore, the processed data storage unit 10 and the child video object data storage unit 9 may be configured to share the same storage device.

[0176] Next, in the image embedding production unit 31, the image embedding or fusion production process is performed similarly to the function unit with the same number shown in Fig. 10. Note that the specific processing contents of this image embedding or fusion production process have already been explained in the first to third embodiments, so a detailed explanation will be omitted here.

[0177] Furthermore, in the video processing, editing and rendering unit 12, similar to the functional units of the same numbers in Figures 1 and 10, the actual processing, editing and rendering of the video is carried out based on the results of the video embedding or fusion performance, so a detailed functional explanation will be omitted here.

[0178] However, the automatic image production device 80 in this embodiment must execute the above-described processing in each functional unit in almost real time. Therefore, depending on the processing capacity of the automatic image production device 80 itself, it may be difficult to execute each of the above-described processes completely without delay. In such a case, taking into consideration the processing capacity of the automatic image production device 80 and the priority of each data and process, it is acceptable to omit some of the predetermined processing steps or the functional units themselves that handle the corresponding processes.

[0179] For example, with regard to video embedding or fusion presentation, it may be limited to the selection of a child video object to be embedded in a parent video, the embedding or fusion position of the target child video object in the parent video scene, and the specification of the anteroposterior or overlapping relationship between the parent video object and the target child video object at that position, and the video presentation may be limited to the subject of video processing and editing processing only on the child video object side.Furthermore, it may be a video presentation that omits all video processing and editing procedures, and only superimposes or embeds a specified child video object or a part thereof into a parent video, and only specifies the screen position in the parent video scene where the specified child video object is to be superimposed.

[0180] Note that, particular care must be taken when applying this embodiment to an AR / MR imaging device using a see-through type HMD.

[0181] That is, if the HMD 44 that displays the AR / MR images is a video see-through type, as described above, both the parent image and the child image object that is embedded or fused into it are video images, so it is naturally possible to freely process and edit both images. However, if the HMD 44 is a so-called see-through type, the actual image that the user 60 sees directly with the naked eye through the see-through display of the HMD 44 becomes the parent image, so it is naturally not possible to process or edit this parent image. Therefore, the image processing or editing process for embedding or fusion of images is limited only to the child image object side. In such a case, the image embedding or fusion performance inevitably has to be simplified or partially omitted as described above.

[0182] In FIG. 12, functional units other than those described above have the same functions as the functional units with the same numbers in FIG. 1 or FIG. 10, so detailed functional description will be omitted here.

[0183] Incidentally, in the present embodiment described above, similarly to the first embodiment, the automatic image production device 80 does not need to be configured with divided functional blocks as shown in Fig. 10. For example, it is perfectly acceptable to use an information processing device such as AI to comprehensively execute the processing in each of the functional units.

[0184] Finally, the parent video, which has been subjected to the embedding or fusion processing of the child video object through each of the above-described processes, is displayed on a specified display of the HMD 44 worn by the user 60 via the AR / MR video playback unit 43. Naturally, if the HMD 44 is of a see-through type, then what is displayed on the see-through display is only the child video object, i.e., the virtual video, which has been subjected to the video embedding or fusion processing applied to the specified AR / MR as described above.

[0185] Next, the processing of the automatic image production device 80 in this embodiment will be described. Fig. 13 is a flowchart of the processing in this embodiment. The processing in this embodiment is basically executed in the same processing flow as the processing described in Fig. 10.

[0186] In FIG. 13, first, in S401, the automatic video production device 80 reads all child video object data, that is, the video data of the child video objects and the cast data linked to the child video objects, from a predetermined data storage unit.

[0187] Next, in S402, playback of the parent image, i.e., the real-time actual visual image, starts (in the case of a see-through HMD, this corresponds to the start of the user's naked eye stereoscopic vision), and the automatic image production device 80 then reads in the image data of the parent image (real-time actual visual image) obtained from the external world shooting camera 19 in the next S403.

[0188] Then, in the next step S404, objects such as people and objects that newly appear in the current parent video scene are identified and extracted as individual parent video objects, and a specified identifier such as an object name or ID number is added to each extracted parent video object and child video object to distinguish them from other video objects.

[0189] In the next step S405, the content of the parent image, i.e., the image actually seen in real time, is decoded and digitized based on the type, position, and movement of each parent image object identified and extracted in S404, as well as information history such as the decoding results and scenario data regarding past images obtained up to that point.

[0190] In the next step S406, the same process as S306 in Fig. 11 is performed to obtain predetermined detection data. In the next step S407, the same process as S307 in Fig. 11 is performed to estimate a predetermined viewer situation. Of course, as in the above-mentioned embodiments, S406 and S407 may be skipped or the processing steps themselves may be omitted as necessary.

[0191] Then, in the next step S408, it is determined whether or not to embed or fuse the child video object into the current parent video cut, i.e., the image actually viewed by the user. If the determination is "Yes," the process proceeds to S409 and subsequent steps, which will be described later. On the other hand, if the determination is "No," steps S409 to S412, which will be described later, are skipped and the process jumps to S413.

[0192] Then, in the next step S409, a child video object to be fitted into the parent video cut is selected from among a plurality of child video objects. Then, in the next step S410, the screen position in the parent video cut where the child video object is to be fitted is specified.

[0193] Furthermore, in the next step S411, in order to blend the target child video object naturally and seamlessly into the target parent video cut at the designated screen position as naturally as possible, a "video processing and editing scheme" is constructed which specifies the specific video processing and editing content and procedures to be applied to the child video object or the parent video cut. Hereinafter, the series of processes from S409 to S411 will be referred to as "video embedding fusion production" processing.

[0194] Next, in S412, the image processing, editing, and rendering processes are actually performed based on the "image processing and editing scheme." As explained in the first to third embodiments, these processes are performed by making full use of predetermined digital image processing technologies such as computer graphics technology, 3D morphing technology, and shading technology.

[0195] However, in this embodiment, the "image embedding fusion production" processing process and the image processing, editing, and rendering processing processes must be executed almost in real time. Therefore, depending on the processing capacity of the automatic image production device 80 itself, it may be difficult to execute each of the above processing processes completely without delay. In such cases, as described above, some of the processes may be simplified or omitted, taking into consideration the processing capacity of the automatic image production device 80 and the priority of each of the data or processing steps.

[0196] Then, in the next step S413, the results of the processing obtained in each of the above processing steps, i.e., the selection of the parent video cut into which the child video is to be embedded or merged and the child video object to be embedded or fused therein, the result of specifying the embedding or fusion screen position within the parent video cut, or various data related to the ``video processing editing scheme,'' and even the parent video data into which the child video object, i.e., the virtual video has been embedded or fused, are written or updated as necessary to a specified data storage unit.

[0197] In the next step S414, the parent image into which the virtual image has been embedded or fused, that is, the real-time actually visually recognized image, is reproduced. Note that the order of steps S413 and S414 may be reversed, or both steps may be performed simultaneously.

[0198] Finally, in S415, it is determined whether or not to end the embedding or fusion process of the child image object in the parent image, i.e., the AR / MR virtual image display by the HMD or the like. If the determination is "No," the process returns to S403 and the series of processing flows is repeated. On the other hand, if the determination is "Yes," the series of processing processes ends.

[0199] As described above, in this embodiment, an AR / MR imaging device can be realized that can embed or blend a predetermined virtual image into an image actually seen by the user in a natural, seamless manner.

[0200] Furthermore, when an automatic video production device such as that of this embodiment is used for purposes such as embedding commercial videos into an AR / MR display screen, as in embodiments 1 to 3, it is possible to count the number of times or the time that the target commercial video is embedded or blended into the parent video, and depending on the counting results, it is possible to combine a mechanism for restricting the provision of commercials or for charging and billing a specified commercial fee to the commercial sponsor.

[0201] FIG. 14 shows an example of an AR imaging device using an HMD that employs the automatic image production device according to this embodiment.

[0202] 14, 55 indicates a scene of a performance site where a person (e.g., the idol (A)) 61 is dancing on a special stage with a bar counter installed on the side, and indicates the real world. 44 indicates a schematic external view of the eyeglass-type HMD worn by the user as seen from the user's side, and the user views the dance performance scene (3D image) of the idol (A) 61 with the naked eye through the left-eye see-through display 45L and right-eye see-through display 45R of this HMD 44.

[0203] FIG. 14 shows an example of an implementation in which the automatic image production device of this embodiment superimposes an image of a specified soft drink plastic bottle as a commercial image in such a situation.

[0204] First, in the above situation, the automatic image production device determines an image fusion production that can be expected to create a natural and certain commercial appeal effect by superimposing an AR image of the PET bottle 46, which is the subject of the commercial, on a bar counter right next to the idol (A) 61.

[0205] Next, the automatic image production device determines the three-dimensional position where the target plastic bottle should be placed in real world 55, and calculates the image display position of plastic bottle 46 that corresponds to that three-dimensional position on each of see-through displays 45L, 45R.

[0206] That is, the depth distance between the target plastic bottle 46 and the user is calculated from the three-dimensional position where the target plastic bottle 46 is virtually placed in the real world 55, and then the binocular disparity d at which the user perceives the depth distance is calculated. Then, an image of the plastic bottle 46 shifted from the other by the binocular disparity d is displayed with AR superimposition on the left eye see-through display 45L and the right eye see-through display 45R, respectively.

[0207] In this way, by superimposing an image of the plastic bottle 46 with a predetermined binocular disparity d on the left and right see-through displays of the HMD in an AR-overlapping manner, the user can perceive the soft drink plastic bottle that is the subject of the commercial as if it were placed on a counter right next to the idol (A) in the real world 55.

[0208] Furthermore, the direction and shape of the shadow of the plastic bottle projected onto the bar counter can be calculated using a specified calculation means based on the direction of incidence of lighting in the real world 55, and this shadow can also be superimposed using a specified shading technology, providing a more advanced image fusion effect, thereby enabling the AR / MR display to produce more realistic images.

[0209] As described above, according to this embodiment, an automatic image production device compatible with AR / MR can be provided. EXAMPLES

[0210] In the fourth embodiment, the real-time visual image in an AR / MR imaging device such as an HMD is used as the parent image into which the virtual image is embedded or merged. Therefore, a series of processing processes such as the "image embedding fusion performance" process for embedding or merging the child image object that becomes the virtual image, the subsequent image processing, editing, and rendering processes, and further the playback and display of the processed image must be executed almost in real time. However, a certain time lag inevitably occurs between the moment when the user visually recognizes the real visual image and the display of the virtual image that is embedded in the real visual image through a series of processing processes, which accompanies the execution of the series of processing processes. Therefore, in an AR / MR imaging device such as an HMD, if an advanced image fusion performance that is rational and natural, such as that introduced in FIG. 7, is to be executed, depending on the processing capacity of the automatic image performance device, it may be difficult to execute the processing with a minute time lag that is felt almost in real time by the user.

[0211] This embodiment describes an embodiment of an AR / MR-compatible automatic image production device that is capable of such advanced image embedding or fusion processing.

[0212] Fig. 15 is a block diagram of the automatic image production device in this embodiment. In Fig. 15, the same reference numbers are used for functional units that have the same functions as those in Fig. 1 or 12.

[0213] The automatic video production device 80 in this embodiment has almost the same functional section configuration as that in Fig. 12. The difference from Fig. 12 is that a video cut estimation section 53 has been newly added. Therefore, detailed explanations of each functional section other than this video cut estimation section 53 will be omitted.

[0214] In addition, the external world capturing camera 19 equipped in the automatic image production device 80 in this embodiment has the function of capturing images of the external world with a wider field of view that includes the actual field of view of the user 60, and the function of defining (framing) the actual visual field of the user 60 within that capturing field of view.

[0215] In FIG. 15, real-time outside image data captured by an outside image capturing camera 19 is read into an automatic image production device 80 as parent image data.

[0216] The video cut estimation unit 53 has a function of estimating parent video cuts from the present time to a predetermined time into the future, for example, a few seconds to tens of seconds, i.e., future visible video cuts that the user 60 is likely to actually view from the present time until the predetermined time has elapsed, based on the parent video data, the video content information of the parent video decoded from the parent video data by the real-time video cut decoding unit 42, the latest scenario information at that time, and detection data regarding the user's gestures and actions, such as the gaze movement of the user 60.

[0217] At this time, the real-time outside world image captured by the outside world shooting camera 19 is a wider field of view image including the actual visual field of the user 60, which is effective for estimating the future visual image based on the gaze movement history of the user 60, etc.

[0218] In addition, this estimated future visual image is always updated to the estimated image that can be judged to be the most correct at that time based on the parent image content information that has been decoded up to that time in the real-time image cut decoding unit 42 and the scenario information that is successively updated to the latest information.

[0219] In this embodiment, the "image embedding fusion performance" processing process as described in embodiment 4 and the image processing, editing, and rendering processing process based on the results of the "image embedding fusion performance" are carried out on the preceding parent image cut estimated as described above, i.e., the future visible image, prior to the current parent image, i.e., the current actual visible image.

[0220] In other words, in this embodiment, a series of image fusion processing steps can be carried out with sufficient time due to advance processing of the estimated future visible image, making advanced image fusion processing possible even in AR / MR imaging devices.

[0221] In the present embodiment described above, similarly to the first embodiment, it is not essential that the automatic image production device 80 is configured with divided functional blocks as shown in Fig. 15. For example, it is perfectly acceptable to use an information processing device such as AI to comprehensively execute the processing in each of the functional units.

[0222] Next, a process of the automatic image production device 80 in this embodiment will be described below. Fig. 16 is a flow chart of the process in this embodiment.

[0223] In Fig. 16, S501 to S505a are exactly the same as S401 to S405 described in Fig. 13, so detailed explanations will be omitted here. Similarly, the following S506 and S507 are exactly the same as S406 and S407 in Fig. 13, so detailed explanations will be omitted here.

[0224] The feature of this embodiment is that in S505b, a parent video cut from the present time to the future for a predetermined time, for example, several seconds to several tens of seconds, i.e., a future actually viewed video that is expected to be actually viewed by the user 60 from the present time until the predetermined time has elapsed, is estimated. If there is a video object on the parent video side that is estimated to newly appear in this future actually viewed video cut, a predetermined identifier is assigned to this estimated parent video object as necessary. Note that the processing order of S505b, S506, and S507 may be interchanged.

[0225] Then, in S508, it is determined whether or not the child video object is to be embedded or blended into the parent video scene. If the determination is "Yes", the process proceeds to S509 and thereafter, which will be described later. On the other hand, if the determination is "No", S509 to S512, which will be described later, are skipped and the process jumps to S513.

[0226] The process contents from S509 to S512 are exactly the same as the process contents from S409 to S412 described with reference to FIG. 13, so a detailed description thereof will be omitted here.

[0227] Then, in the next step S513, a process is performed in which the child video object, i.e., the parent video data that has already been subjected to the embedding or fusion process of the virtual video, is sequentially added to a predetermined data storage unit. Also, as necessary, the process may be performed to write or update the processing results obtained in each processing step, i.e., the parent video cut that is the subject of the child video embedding or fusion and the selection result of the child video object to be embedded or fused therein, the designation result of the embedding or fusion screen position within the parent video cut, or various data related to the "video processing editing scheme," etc., to the data storage unit as necessary.

[0228] Then, in the next step S514, the child video object to be played at that time, that is, the parent video having been subjected to the embedding or blending process of the predetermined virtual video, is played in accordance with the passage of time.

[0229] Then, in the final step S515, it is determined whether or not to end the series of processes for embedding or fusing the child video object with the parent video, i.e., the display of the virtual video in AR / MR. If the determination is "No," the process returns to S503 and the series of processes is repeated. On the other hand, if the determination is "Yes," the series of processes ends.

[0230] In the above explanation of this embodiment, for the sake of convenience, each processing step has been described as being executed sequentially according to the flowchart in FIG. 16. However, in this embodiment, a plurality of processing processes, each of which is made up of several processing steps, are actually executed independently and simultaneously in parallel.

[0231] For example, the process of estimating the future parent video cut from S503 to S507, the process of "video embedding fusion production" from S508 to S512, and the process of processing, editing, and rendering the video based on the results of the "video embedding fusion production" are executed simultaneously in parallel.

[0232] In other words, the automatic image production device 80 successively estimates the latest future parent image from the image actually viewed by the user in real time, while at the same time executing an "image embedding fusion production" process using the future parent image that has already been estimated, and an image processing, editing, and rendering process based on the production results.

[0233] Furthermore, since the parent image playback in S514 is an image that corresponds to the image actually viewed in real time at that time, the parent image playback in S514 and the series of processing steps from S503 to S513 are actually executed in parallel.

[0234] That is, in this embodiment, while playing back the real-time real-view image, which is the parent image (in the case of a video see-through type HMD) or viewing the real image with the naked eye (in the case of a see-through type HMD), a process is simultaneously performed in parallel to embed or fuse a predetermined child image object, i.e., a virtual image, into an estimated future visual image that precedes the image by a predetermined time. By performing such parallel processing, even when a real-time real-view image is used as the parent image to be embedded or fused as in an AR / MR imaging device, a natural virtual image fusion without a sense of incongruity can be performed by advanced image fusion.

[0235] Furthermore, when an automatic video production device such as that of this embodiment is used for purposes such as embedding commercial videos into an AR / MR display screen, as in embodiments 1 to 4, it is possible to count the number of times or the time that the target commercial video is embedded or blended into the parent video, and depending on the counting results, it is possible to combine a mechanism for restricting the provision of commercials or for charging and billing a specified commercial fee to the commercial sponsor.

[0236] As described above, according to this embodiment, an AR / MR-compatible automatic image production device capable of advanced image embedding or fusion processing can be provided. EXAMPLES

[0237] In this embodiment, an automatic broadcast service system for commercial videos will be described as an example of a service using the automatic video production device described in the first to fifth embodiments.

[0238] Fig. 17 is a conceptual diagram of a commercial video fusion service system using an automatic video production device in this embodiment. In Fig. 17, first, a commercial sponsor (e.g., a specified product manufacturer) Company X issues a request to a video content producer / distributor or a manager (company) of the automatic video production device 80, Company Y, which manages the automatic video production device 80, to broadcast a commercial video in a specified video content. In response, Company Y issues a video fusion production command to the automatic video production device 80 that it manages.

[0239] When the automatic video production device 80 receives a video fusion production command, it first reads the video data of the target parent video content from Y's server 82 or an external server 83 such as a cloud connected via a predetermined communication means or network. Note that this parent video content may be general pre-produced video content obtained from a predetermined server as described above, or may be a video actually viewed by the user obtained through an HMD or the like as shown in the fourth and fifth embodiments.

[0240] On the other hand, the automatic video production device 80 receives video data to be advertised as a child video object from a server 81 of company X, which is a commercial sponsor, or an external server 83 such as a cloud connected via a predetermined communication means or network.

[0241] Then, using these parent and child images and the specified viewer-related information provided by the viewer, the commercial fusion production processing is carried out through the processing steps shown in each of the above embodiments, and the parent image that has been processed for commercial fusion is distributed and broadcast to the viewer 60 via a specified display device 14.

[0242] On the other hand, the automatic video production device 80 is connected to a predetermined billing system 84, which counts the number of times and the duration for which the target commercial video is embedded or blended into the parent video content, and depending on the counting result, the provision of commercials is restricted or a predetermined commercial fee is charged and invoiced to the commercial sponsor, Company X.

[0243] If such an automatic CM video provision service system is established, it is possible to always obtain an optimal viewing environment for video content with CMs for both the sponsor and the user with almost no human intervention.

[0244] Although the embodiments have been described above, the present invention is not limited to the above-mentioned embodiments, and various modified examples are included. For example, the above-mentioned embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. It is also possible to add the configuration of one embodiment to the configuration of another embodiment. It is also possible to add, delete, or replace other configurations with respect to part of each embodiment. [Explanation of symbols]

[0245] 1: Video content data reading unit, 2: Video content decoding unit, 3: Video object identification and extraction unit, 4: Viewer biometric data detection unit, 5: Viewer gesture and behavior data detection unit, 6: Surrounding environment data detection unit, 7: Casting data generation unit, 8: Script data generation unit, 9: Child (fusion target) video object data storage unit, 10: Processing data storage unit, 11: Video fusion production unit, 12: Video processing, editing and rendering unit, 20: Control unit, 31: Video embedding production unit, 32: Advance cut video summary decoding unit, 53: Video cut estimation unit, 80: Automatic video production device, 44: Head mounted display (abbreviated as HMD), 60: Viewer or user

Claims

1. An automatic video production method for embedding or merging a predetermined child video object not included in a parent video into a predetermined video scene of the parent video that is a video fusion source, the method comprising: Decoding the video content of a predetermined first video scene in the parent video, and estimating the video content of a second video scene that is broadcast a predetermined time after the first video scene based on the decoding result, The automatic video production method is characterized by comprising a video cut estimation means for generating processed data, which is predetermined parent video estimation data, which is a data of the parent video content estimation result in the second video scene in a predetermined description format.

2. The automatic image production method according to claim 1, The automatic video production method is characterized by comprising a video object identification and extraction means for identifying and extracting a specified parent video object and a child video object visualized within the parent video from other video objects based on the processed data, and adding a specified object identifier.

3. The automatic image production method according to claim 2, For each video object identified and extracted by the video object identification and extraction means, a physical attribute item that defines a physical classification, attribute or feature of the video object from a predetermined viewpoint, or A social attribute item that defines the social classification, attributes, or characteristics of the video object from a given perspective, or An item specifying the category or characteristics of the visual appearance of the image object; or An item specifying the classification or characteristics of the non-visible state of the video object; or An item that clearly indicates the mutual relationship or connectivity between the video objects; The automatic video directing method further comprises a cast data generating means for generating processing data which is predetermined cast data obtained by converting predetermined information related to the video into data or parameters in a predetermined description format.

4. The automatic image production method according to claim 2, Among the parent video objects identified and extracted by the video object identification and extraction means, a predetermined parent video object or child video object visualized in a predetermined parent video scene in the parent video is An item related to the three-dimensional position of the parent video object within the parent video scene, or An item related to the three-dimensional orientation or posture of the parent image object, or Items related to the parent video object's lines, facial expressions, gestures, or actions, or a predetermined constraint imposed on the parent video object or the child video object; or Constraints imposed on the entirety of the given parent video scene, including designation of a video area within which the child video object is allowed to be embedded or blended; The automatic video production method includes a script data generating means for generating processing data, which is script data obtained by digitizing or parameterizing predetermined information related to a video in a predetermined description format.

Images