Information processing device, information processing method, and program

Abstract

Provided is an information processing device comprising a story generating unit for performing, with respect to an automatically produced content, a process for requesting an external apparatus to record material content of which a story is composed.

Description

[Technical field]

[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to a technical field of automatic production of content using images and audio. [Background technology]

[0002] Patent Document 1 listed below discloses a technique for producing video content by connecting unit videos as material. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-130597 A Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, due to the miniaturization of recording devices such as imaging devices and audio recording devices, their incorporation into portable devices, and the widespread use of various devices, it has become easier and more common to capture images (images are a general term for videos and still images) and record audio. In addition, due to the diversification of recording media such as memory cards and the widespread use of cloud storage using network communication, the manner in which data such as photos, videos, and audio is stored has also become more diverse. In such an environment, for example, it has become relatively easy for a server device to acquire images of customer users, and it is conceivable to create content such as short films starring customer users as a service. Furthermore, due to recent advances in AI processing, it is conceivable to accurately collect images and other materials and automatically create content that is desirable for customer users.

[0005] Here, when considering that the server device providing the service automatically produces content and provides it to the customer, when the server device, for example, collects image material of the customer user and automatically produces content with that user as the main cast, it is not necessarily able to obtain image material sufficient to construct a story. Therefore, an object of the present technology is to make it easier to obtain appropriate materials when automatically producing content to be provided to users. [Means for solving the problem]

[0006] An information processing device according to the present technology includes a story generation unit that performs processing for requesting an external device to record material content that constitutes a story for automatically produced content.

[0007] Consider the automatic production of content with a certain story using multiple material contents. Material contents are images (videos and still images) and audio that make up each scene that makes up the story. In this case, a process is performed to request an external device to record the images and audio that are to be used in the story. Recording includes, for example, image recording by capturing images and audio recording. A variety of external devices are envisioned, including devices owned by the customer user, other devices connected to a network, public devices, and devices provided for the operation of the automatic content production.

[0008] In the information processing device related to the above-mentioned present technology, the story generation unit may set the story of the content to be automatically produced, select the material content to be used, and perform a story setting process to determine whether there is a shortage of material content, and if it determines that there is a shortage of material content, perform a process to request an external device to record the material content. By performing a story setting process and selecting material contents according to the story, it is possible to determine whether there is a surplus or deficiency. If there is a deficiency, a request is made to record the material contents.

[0009] In the information processing device related to the present technology described above, the story generation unit may perform a timing determination process for a user who is considered to be a customer, and in accordance with the timing determination process, perform a process of requesting an external device to record material content. The opportunity for automatic content production for a user who is a customer of the automatic content production service is determined based on the user's event schedule, life stage, etc. Then, a process is performed to request the recording of material content according to the determined time.

[0010] In the information processing device according to the present technology described above, it is considered that the story generation unit determines a user's preference based on user information, and performs processing to request recording of material content based on the determined preference information. For example, the preference determination is made based on user information about the user himself / herself who is regarded as a customer, his / her family and friends, and a request for recording of material content is made based on this.

[0011] In the information processing device according to the present technology described above, it is considered that the story generation unit performs processing to determine a user's preference based on user information, and to select material content based on the determined preference information. For example, the preference determination is made based on user information about the user himself / herself who is regarded as a customer, and his / her family and friends, and material content is selected based on this.

[0012] In the information processing device according to the present technology described above, it is considered that the story generation section performs processing for requesting recording of material content on the basis of event information relating to a user. For example, the appropriate scene or the like is determined according to the type of event information of a user who is considered to be a customer, and a recording request is made for the appropriate material content.

[0013] In the information processing device according to the present technology described above, it is considered that the story generation section performs a process of selecting material content based on event information related to a user. For example, the appropriate scene or the like is determined according to the type of event information of the user who is regarded as a customer, and suitable material content is selected.

[0014] In the information processing device according to the present technology described above, it is conceivable that the timing determination process may involve performing timing determination based on a life stage of a user. For example, the life stages of the user are determined as birth, school entrance, graduation, employment, marriage, birth of children, children's growth, retirement, etc., and appropriate material content is selected and recording requests are made accordingly.

[0015] In the information processing device relating to the above-mentioned present technology, the story generation unit may generate recommendation data indicating the details of the material content for which recording is requested, based on the event that caused the timing determination process to determine that it is the time to request recording, and content of other users regarding the same event. For example, the event that is the cause of the determination that the time to request recording in the timing determination process is an event in the user's life stage (admission, graduation, employment, marriage, etc.), various events, etc. When it is determined that the time to request recording is due to such an event, data is generated that recommends the contents of the scene to be photographed, etc., based on the content of the same event for other people.

[0016] In the information processing device related to the present technology described above, the story generation unit sets a story for content to be automatically produced for a user who is considered a customer, selects material content to be used from a database corresponding to the user, and performs a story setting process to determine whether there is a shortage of material content.If it is determined that there is a shortage of material content, it searches for material content from other databases, and if it is further determined that there is a shortage of material content, it performs a process of requesting an external device to record the material content.

[0017] For example, a story setting process is performed, and material contents matching the story are searched for in a database corresponding to the user, and if the content is insufficient, other databases, for example, databases corresponding to other users or public databases, are searched for. If the content is still insufficient, a request to record the material contents is made.

[0018] In the information processing method according to the present technology, an information processing device performs processing for requesting an external device to record material content constituting a story for automatically produced content, thereby generating an opportunity to obtain necessary material content.

[0019] A program according to the present technology causes an information processing device to execute such an information processing method, thereby facilitating realization of the information processing device according to the present technology. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is an explanatory diagram of a system configuration according to an embodiment of the present technology; [Diagram 2] FIG. 2 is an explanatory diagram of a functional configuration related to automatic content production according to an embodiment; [Diagram 3] 1 is a block diagram of a configuration of an information processing device according to an embodiment; [Figure 4] FIG. 2 is an explanatory diagram of an example of a story structure of automatically produced content according to the embodiment; [Diagram 5] 4 is an explanatory diagram of the operation of a function relating to automatic content production in the server device of the embodiment; FIG. [Figure 6] FIG. 2 is an explanatory diagram of an example of a workflow executed in the system according to the embodiment. [Figure 7] FIG. 4 is an explanatory diagram of an example of relevance presentation information according to an embodiment; [Figure 8] FIG. 4 is an explanatory diagram of an example of relevance presentation information according to an embodiment; [Figure 9] FIG. 4 is an explanatory diagram of an example of relevance presentation information according to an embodiment; [Figure 10] FIG. 4 is an explanatory diagram of an example of relevance presentation information according to an embodiment; [Figure 11] 13 is a flowchart of an automatic production process according to an embodiment. [Figure 12] 11 is a flowchart of a story setting process according to an embodiment. [Figure 13] 13 is a flowchart of a material content complementing process according to an embodiment. [Figure 14] 13 is a flowchart of a material content complementing process according to an embodiment. [Figure 15] 13 is a flowchart of a material content complementing process according to an embodiment. [Figure 16] 13 is a flowchart of a material content request process performed in advance according to an embodiment. [Figure 17] 13 is a flowchart of a material content complementing process according to an embodiment. [Figure 18] 13 is a flowchart of a material content complementing process according to an embodiment. [Figure 19] 13 is a flowchart of a material content evaluation process according to an embodiment. [Figure 20] 13 is a flowchart of a feedback process according to an embodiment. [Figure 21] 11 is a flowchart of a process for setting a story according to an embodiment. [Figure 22] 11 is a flowchart of a feedback handling process according to an embodiment. [Diagram 23] 13 is a flowchart of a material content management process according to an embodiment. [Figure 24] 13 is a flowchart of a material content management process according to an embodiment. [Diagram 25] 13 is a flowchart of a material content management process according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] The embodiments will be described below in the following order. <1. System configuration> <2. Overview of Automated Content> <3. Content creation process> <4. Automatic production process including content completion> <5. Pre-production recommendations / complementary processing> <6. Evaluation and feedback of material content> <7. Management of negative / positive content> <8. Summary and variations>

[0022] In this disclosure, the following terms are used to distinguish between "contents". Automated content In the present disclosure, for example, content to be provided to a user as a customer is automatically produced by an information processing device. The automatically produced content to be viewed by a user is called "automatically produced content", but in the embodiment, since an example of "automatically produced content" that is automatically produced for the purpose of inspiring a customer user or the like is given as the "automatically produced content", for convenience of explanation, it is mainly called "moving content". "Moving content" is one specific example of "automatically produced content". Automatically produced content comes in a variety of forms, including content including video / audio, content such as a slideshow of still images, content including still images and audio, audio-only content, content including video, still images and audio, and content including text data.

[0023] Material content Automatically produced content is created by combining multiple video materials, still image materials, audio materials, text materials, etc. Each piece of content data, such as video materials, still images, audio materials, and text materials, is called "material content." Any type of content can become material content. · Candidate content This refers to material content that is selected as a candidate for use as images and sounds for a certain scene when automatically produced content is created. For example, from a large amount of material content, several candidate contents that are likely to be suitable for a certain scene are extracted, and the material content to be used for that scene is selected from the candidate contents.

[0024] Negative and positive content Among the material contents, those that are determined to have scene contents such as optimistic, positive, and progressive are called "positive contents." Among the material contents, those that are determined to have scene contents such as pessimistic, negative, or regressive are called "negative contents." However, this "positive content" and "negative content" are not absolute divisions, and some judgment criteria may be determined according to the scene content type, or according to the customer user, or according to the time / era, or according to common sense, social conditions, etc. In this technology, at least material content judged as "negative content" is managed separately from other material content.

[0025] In addition, in this disclosure, the term "image" may refer to either a moving image or a still image, and is used as a general term for both.

[0026] <1. System configuration> FIG. 1 shows an example of a system configuration including a server device 10. In the embodiment, an example will be described in which an information processing device constituting the server device 10 automatically creates moving content for a customer user using a large number of image materials, etc. from a large number of users, and provides the moving content to the user. In addition, the material content will be described as an example mainly consisting of video data (including audio). In other words, it is assumed that the automatically produced moving content will be an example of a movie produced by editing multiple videos.

[0027] A server device 10 in FIG. 1 is an information processing device capable of communicating with various devices via a network 11 such as the Internet, and functions as, for example, a so-called cloud server.

[0028] The server device 10 can store a large amount of material content and created moving content in the storage 10a under its management. The storage 10a indicates a storage accessible by the server device 10, and may be a storage device mounted inside an information processing device serving as the server device 10, or may be a storage device connected via a LAN (Local Area Network) or the like. Also, a storage device located away from the server device 10, or a storage device on the user side may be used as the storage 10a referred to here. That is, if server device 10 is a storage device from which material content can be collected for the automatic generation of moving content, there is a possibility that it corresponds to storage 10a shown here.

[0029] The server device 10 is also capable of storing material contents and the like in a remote storage 12 . The remote storage 12 is shown as a storage device with lower accessibility compared to the storage 10a. The remote storage 12 may be a storage device within the server device 10 or an external storage device. A detailed example will be described later. The remote storage 12 may be a storage device that cannot be accessed physically or through access authority management from user-side equipment, such as the illustrated computer device 3, tablet terminal 4, mobile terminal 5, etc.

[0030] Such a server device 10 may be a server that provides a cloud storage service on a network, a server that provides a social networking service (SNS), or a platform server that provides a comprehensive service.

[0031] The network 11 may be, for example, the Internet, a home network, a LAN, a satellite communication network, or any other type of network. Various devices can upload image data and the like to the server device 10 via the network 11 and download various information from the server device 10 . In addition, for example, communication using FTP (File Transfer Protocol) may be adopted as a part or the whole of the network 11. In addition, short-range wireless communication such as Bluetooth (registered trademark), WI-FI (Wireless Fidelity: registered trademark), or NFC (Near Field Communication: registered trademark), or infrared communication may be adopted. In any case, the network 11 may be any network capable of data communication.

[0032] In Fig. 1, devices capable of taking and recording images on the user side are shown as a camera 2, a computer device 3, a tablet terminal 4, a mobile terminal 5, a camera-equipped aircraft 6, a camera-equipped vehicle 7, an agent device 8, and a surveillance camera 9. These are merely examples of devices capable of taking and recording images, and other devices are also possible.

[0033] The camera 2 is an imaging device that can capture still images and videos and record or transfer the captured images as image data. It is expected that the camera 2 will be used by many users, and the captured image data (videos and still images) will be stored in the storage 10a or the like, so that the server device 10 will be able to use the image data as material content.

[0034] The computer device 3 is exemplified by a notebook personal computer, a desktop personal computer, etc. A tablet terminal 4 and a mobile terminal 5 such as a smartphone can also be considered as an embodiment of the computer device. In the case of these computer devices 3, tablet terminals 4, mobile terminals 5, etc., it is assumed that image data captured by the built-in camera, image data transferred from other devices such as the camera 2, image data generated as composite images or computer graphics (CG), image data as images in video chats, image data shared with family and friends, etc. are stored in the storage 10a, etc., and can be accessed by the server device 10.

[0035] The camera-equipped aircraft 6 is assumed to be an aircraft equipped with a camera such as a drone. It is assumed that image data captured by the camera-equipped aircraft 6 is stored in the storage 10a or the like and can be accessed by the server device 10. Drones and other devices are expected to be owned by users, public property, or owned by operators of inspirational content provision services and used to photograph customer users in order to collect material content.

[0036] The camera-equipped vehicle 7 is, for example, a vehicle such as an ordinary automobile, and is equipped with a camera such as a drive recorder. For example, by providing an image from a drive recorder of a vehicle owned by a user to the server device 10, it is assumed that image data captured by the camera-equipped vehicle 7 is stored in the storage 10a or the like and can be accessed by the server device 10.

[0037] The agent device 8 is an interface device for providing services such as information provision and home appliance control in the home, and is assumed to be capable of taking pictures by being equipped with a camera. It is assumed that image data obtained by taking pictures with the agent device 8 is stored in the storage 10a or the like and can be accessed by the server device 10. Here, an example is given of an agent device 8, but it is assumed that, for example, a housekeeping robot or a pet-type robot used in the home may also have a photographing function, and image data captured by these robots may be stored in storage 10a or the like and made accessible by server device 10.

[0038] The surveillance camera 9 may be installed in a public place, or may be installed in a private space by a general user. It is assumed that image data captured by such a surveillance camera 9 is stored in a storage 10a or the like and can be accessed by the server device 10.

[0039] The various devices as described above may be devices capable of obtaining material contents used by the server device 10, or devices capable of providing a user interface.

[0040] FIG. 2 illustrates the relationship between the server device 10 and various devices. The various devices shown in FIG. 1 may function as the material content provider 20 in FIG.

[0041] For example, the above-mentioned devices such as the camera 2, the computer device 3, and the surveillance camera 9 can be sources of material content that the server device 10 can use since they can capture or generate content such as video data. In other words, images obtained by various devices can become material content for creating moving content such as a customer user's personal movie. These are merely examples, and other possibilities are also possible, but examples of images that could become material content are as follows:

[0042] - Images taken or recorded by the customer user and uploaded to the cloud Images taken by the service provider upon request Images provided by the customer user's local data release - Images taken and uploaded by the Customer User's friends, family, etc. - Images taken by robots, drones, in-vehicle cameras, etc. Images from local cameras, surveillance cameras, etc. Social media images of yourself, your friends, followers, etc. Images during video chats and other communications - Images created using image processing, such as CG images or fake images Public images and images that are allowed to be used freely

[0043] These images are stored, for example, in the storage 10a or the remote storage 12 in a state where they can be managed as images corresponding to individual users. When a certain user A becomes a customer and produces moving content, images associated with that user A become usable material content. Each image data may be stored together in association with a user's personal identification information, or a single image may be assigned multiple user identification information so that a single image can be used for multiple users.

[0044] The camera 2, the computer device 3, the mobile terminal 5, etc. are devices equipped with UI functions such as display and operation input, etc. Therefore, these devices can function as the UI unit 22 when displaying to the user the moving content and various images provided by the server device 10, or when accepting user operations.

[0045] FIG. 2 also shows an example of the functional configuration of the server device 10. The server device 10 includes an evaluation unit 15, a story generation unit 16, a learning unit 17, and a memory management unit 18 as functions related to the automatic production process of moving content.

[0046] The evaluation unit 15 is a processing function that sets an evaluation value for each piece of material content that is stored in, for example, the storage 10a and that the server device 10 can access. The story generation unit 16 is a function for automatically producing moving content by carrying out story setting processing for moving content for a customer user, selection processing for material content corresponding to the story, editing processing for the selected material content, and the like. The learning unit 17 is a function that receives feedback from the user and performs learning processing to adapt to the individual user. The storage management unit 18 performs storage management of material contents, in particular, storage management of negative contents and positive contents. Specific processing examples of these functions will be described later, but high-quality processing can be achieved by performing AI processing for each function, for example.

[0047] FIG. 2 also shows the function of a user information providing unit 21. This user information providing unit 21 may be a function built into the camera 2, computer device 3, tablet terminal 4, mobile terminal 5, etc., or may be a separate device or sensor device. It may also be a wearable device that can obtain user information at each point in time (location information indicating the user's location, vital information, etc.).

[0048] User information is expected to include various types of information, such as the user's age, gender, and other attribute information, hobbies, the user's face, family composition, ages of family members, information on friends, schedule information, event information (plans, participation history, SNS information, etc.), the user's camera history, vital information, and location information.

[0049] Such user information may be managed in association with an individual user, or associated with material content such as image data, etc. For example, it may be added to and associated with image data as metadata. For example, attribute information such as the user's age and sex, hobbies, the user's face, family structure, ages of family members, information on friends, schedule information, event information, camera history, etc. may be associated with an individual user. The server device 10 manages such user information for each user as information for producing moving content appropriate for the user. In terms of information at the time of shooting, the user's age at the time of shooting, family composition at the time of shooting, information on friends, etc., schedule information and event information corresponding to the shooting content (information on what event the image was taken at), vital information at the time of shooting, location information at the time of shooting, etc. can be user information associated with the image. Such user information is stored as metadata for each material content, for example, or managed as a separate information file corresponding to the material content. This allows the server device 10 to use user information corresponding to each material content.

[0050] The user information provided by the user information providing unit 21 may include not only directly detected information but also information obtained by, for example, the result of image analysis or the analysis of a plurality of pieces of sensing data. For example, analysis of captured image data may provide information on the user's appearance, facial expression, complexion, and emotions inferred from these. Furthermore, when the heart rate, brain sensing information, blood pressure, amount of sweat, etc. are measured as vital sensors, the user's emotions can be inferred from these and used as user information.

[0051] Information about human relationships at the time of photographing is also useful. For example, information of types such as an image taken by the target customer user himself, an image taken by a friend of the customer user, an image taken by a family member of the customer user, and an image objectively taken by a surveillance camera can be user information associated with the image.

[0052] The user information obtained by the user information providing unit 21 as described above is provided to the device which serves as the material content providing unit 20 and added to the image data as metadata, or provided to the server device 10 and managed in association with the individual user or the material content.

[0053] Next, an example of the hardware configuration of an information processing device 70 that can be configured as the server device 10, the computer device 3, the tablet terminal 4, the mobile terminal 5, or the like is shown in FIG.

[0054] A CPU (Central Processing Unit) 71 of the information processing device 70 executes various processes according to a program stored in a ROM (Read Only Memory) 72 or a program loaded from a storage unit 79 to a RAM (Random Access Memory) 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes as appropriate. The CPU 71, the ROM 72, and the RAM 73 are interconnected via a bus 74. To this bus 74, an input / output interface 75 is also connected.

[0055] The input / output interface 75 is connected to an input unit 76 including an operator and an operating device. For example, the input unit 76 may be various types of operators or operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. An operation by the user is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71.

[0056] Further, the input / output interface 75 is connected, either integrally or separately, to a display unit 77 formed of a liquid crystal panel (LCD: Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or the like, and an audio output unit 78 formed of a speaker, or the like.

[0057] The display unit 77 is a display unit that performs various displays, and is configured, for example, by a display device provided in the housing of the information processing device 70, a separate display device connected to the information processing device 70, or the like. The display unit 77 displays images for various types of image processing, moving images to be processed, etc., on the display screen based on instructions from the CPU 71. Furthermore, the display unit 77 displays various operation menus, icons, messages, etc., that is, as a GUI (Graphical User Interface), based on instructions from the CPU 71.

[0058] The input / output interface 75 may be connected to a storage unit 79 configured with a hard disk or solid-state memory, or a communication unit 80 that performs communication using various communication methods.

[0059] The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, and the like. When the information processing device 70 is considered as the server device 10 , the communication unit 80 has at least a function of performing communication via the network 11 .

[0060] A drive 82 is also connected to the input / output interface 75 as required, and a removable recording medium 81 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon. The drive 82 allows data files such as image files and various computer programs to be read from the removable recording medium 81. The read data files are stored in the storage unit 79, and images and sounds contained in the data files are output on the display unit 77 and the sound output unit 78. Furthermore, the computer programs and the like read from the removable recording medium 81 are installed in the storage unit 79 as necessary.

[0061] In this information processing device 70, for example, software for the processing of the present disclosure can be installed via network communication by the communication unit 80 or via a removable recording medium 81. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

[0062] When the information processing device 70 is assumed to be the server device 10, the functional configuration described in FIG. 2 is constructed in the CPU 71 of the information processing device 70 by software (application program).

[0063] <2. Overview of Automated Content> The server device 10 produces moving content as a personal movie for a certain customer user as automatically produced content. The outline of the story structure of this moving content will be explained.

[0064] FIG. 4 shows an example of a story structure for moving content and corresponding variations. Here, we give an example of a story structure that includes a beginning, development, twist, and conclusion, and has the following chapters: CH1, CH2, CH3, CH4, and CH5.

[0065] Chapter CH1: Past Life Here, scenes showing the things the user does every day are placed, such as baseball practice, piano practice, commuting to work, friendships, etc.

[0066] Chapter CH2: An incident occurs Place scenes that show what caused you to stop (or be unable to) do something that you used to do on a daily basis, or the events that led to the change. For example, events such as injuries, new friendships, transfers, graduations, etc.

[0067] Chapter CH3: A digest of what happens when life returns to normal For example, placing negative content. These include scenes showing the struggles of rehabilitation from injury, bad play, hardships, lonely days, etc.

[0068] Chapter 4: Current Daily Life Include scenes that show the current daily life, having escaped from the negative situation. For example, returning to baseball practice, devoting oneself to piano practice, being transferred again, reuniting with friends, etc.

[0069] Chapter CH5: Main Scene Place exciting (moving) scenes in the story. For example, a scene where they return to a game and perform well, a piano recital, a career achievement, the development of friendships, etc.

[0070] The above is just one example, but it is conceivable to generate personal moving content for customer users by including an introduction, development, twist, and conclusion, and featuring moving scenes.

[0071] <3. Content creation process> For example, the flow of automatic production of moving content having such a story structure will be described. 5 shows an overview of the process flow of the evaluation unit 15, learning unit 17, and story generation unit 16 as functions of the server device 10. It also shows the relationship with the material content provision unit 20, user information provision unit 21, and UI unit 22.

[0072] As described above, the server device 10 is provided with images and sounds that can become material contents, and metadata associated therewith (including user information associated with the material contents), from the material contents providing section 20. In addition, user information is provided from a user information providing unit 21 .

[0073] These material contents and user information are stored and managed in the server device 10 by the function of the memory management unit 18 shown in Figure 2, for example, by associating them with each individual user, and an evaluation value setting process is performed on the stored material contents by the evaluation unit 15.

[0074] The evaluation section 15 sets an evaluation value indicating the relevance of each piece of material content to other pieces of material content, and also sets an individual evaluation value for each piece of material content.

[0075] The relevance evaluation value for a certain material content is a value indicating the relevance of that material content with other material content, and is a value that quantifies the magnitude of relevance, such as causal relationship or chronological relationship.

[0076] The individual evaluation value is a value that evaluates the value of the image or scene of the material content itself, the importance of the contents, etc. For example, if the image content expresses any of the emotions of joy, anger, sorrow, or pleasure, a high evaluation value is set. Furthermore, photographs in which a subject is set as a tracking target are given a high evaluation value as material content for the user of that subject. Additionally, images of unexpected events or events that do not occur in everyday life are highly rated as useful material content for story construction. Images with positive elements or images with negative elements are highly evaluated because they are useful material content in constructing a story as positive or negative content. Information on a user's viewing history, such as what the user liked and watched many times, or what the user never plays, indicates the importance or favorites of the user, so an evaluation value can be set according to the viewing history.

[0077] The evaluation unit 15 sets, for each user who is to become a customer, a relevance evaluation value and an individual evaluation value, for example, as described above, for each piece of material content associated with that user. These settings can be made by comprehensively reflecting the results of scene content determination through image analysis, image quality, user information, and the like.

[0078] Such evaluation values ​​are provided to the story generation unit 16. The story generation unit 16 performs processing to generate personal emotional content for a certain user based on some kind of trigger. For example, the story generation unit 16 sets up a story with the customer user or a family member as the main character, and collects material contents that can be used for each scene constituting the story as candidate contents. Then, for each candidate content, a calculation is performed using an evaluation value (evaluation function calculation), and material contents to be actually used as moving content are selected. The selected material content is then used for editing to generate moving content such as a movie starring the customer user and his / her family.

[0079] The moving content thus produced is provided to the user, and becomes viewable by the user via the UI unit 22.

[0080] The above is the basic flow of automatic content production, and an example of this automatic production timeline is shown in Figure 6. The chronological steps are as follows: Step ST1: Shooting videos and still images as content material Step ST2: Transfer of material content from the user side to the server device 10 (making it available through the server device 10) Step ST3: Storing and managing material content on the server device 10 side Step ST10: Planning a story for moving content Step ST20: Selecting content material according to the story you have set Step ST30: Post-processing the material contents such as image quality processing and scene extraction, and editing the material contents that have been subjected to such processing, such as joining them together. Step ST40: Providing moving content to the user side (viewing on the user side)

[0081] Here, steps ST10, ST20, and ST30 correspond to the process of automatic content production by the story generation unit 16.

[0082] FIG. 6 shows various examples of workflows. The workflows WF1 and WF2 show the process up to collection of material contents.

[0083] Workflow WF1 is an example in which, for example, a user takes a video using a camera 2 or the like, the video data is transferred to server device 10, and a database (hereinafter, database will be referred to as a "DB") for that user is generated or recorded in the DB, thereby making material content for that user available for use.

[0084] Workflow WF2 is an example in which images of a user are collected by a method other than photographing the user, and are managed in a DB for the user in server device 10, making them available for use as material content for the user. In this case, the collected images may be images taken by other users or images collected as public data. This is an example of collecting material content from a wider range than just material content taken by the user himself.

[0085] The workflows WF3 to WF8 show examples of automatic content production by the story generation unit 16.

[0086] The workflow WF3 is started by a trigger such as a request from a user to create moving content or the status of a given material content. First, a story about a certain user is conceived, then the database for that user is referenced and content material is extracted according to the structure of the story.Then, the content material to be used is selected and edited to generate moving content.

[0087] A production request from a user is assumed as a trigger for automatic production of moving content. For example, the user presents information indicating what kind of content they want, such as specifying the theme, title, and specific material content of the moving content, and the server device 10 automatically generates the moving content based on that information. For example, it is conceivable that the user may specify only a portion of the material content that he or she wishes to use in the moving content, and the server device 10 may set a story accordingly, and select material content for the scenes before and after the designated portion to create the moving content.

[0088] Furthermore, the automatic production of moving content can be triggered not only by active instructions from the user as described above, but also by a message from the server device 10 to the user suggesting, for example, "Would you like to produce content like XX?", and then production can begin if a request is made in response. For example, if material content of the user's child's athletic meet is collected based on the collection status of material content, a message such as "Would you like to make a video of your child's athletic meet?" may be sent. Furthermore, if a certain negative content is encountered and corresponding positive content can be obtained, a story like that shown in Figure 4 above can be constructed. In such a situation, it is possible to encourage the user to create content and start automatic creation depending on the user's response. In addition, when there is material content that the user may have forgotten about, such as an image that was taken a long time ago or an image that the user has hardly viewed, it is possible to encourage the user to create content based on the content of the image, and to start automatic creation depending on the user's response. Similarly, if a user has images taken when starting a new activity, or at a place they visited for the first time or a place they haven't visited in a while, the system could encourage the user to create content based on the content of the images, and automatic creation could begin depending on the user's response.

[0089] These messages may be sent via email or SNS messages to users who have registered for the service, or displayed on an application screen on a user's terminal device (computer device 3, tablet terminal 4, mobile terminal 5, etc.) on which an application for providing the service has been installed, or via push notification, etc.

[0090] Furthermore, as a trigger for the automatic production of moving content, it is also conceivable that the production of moving content can be started automatically without waiting for a user request, for example by automatically setting a theme or title from the content material.

[0091] Similarly, workflow WF4 designs a story about a certain user in response to a trigger such as a request from the user, and then references the user's DB to extract material content according to the structure of the story, but this is an example of a case in which material content is insufficient. In such a case, as a material content supplement process, images of the missing scenes are collected by instructing the user to shoot or by performing some kind of automatic shooting. After that, when the necessary material content is collected, an editing process is performed to generate moving content. An example of an instruction to the user to shoot may be a message to the user saying, "Please shoot a scene like this." An example of an instruction to automatically shoot may be an instruction to a robot or home device owned by the user to shoot a specified scene, or an instruction to a shooting company's device.

[0092] Workflow WF5 is an example of automatic production of moving content based on event information. It checks the user's attribute information, schedule data, SNS information, etc., detects the occurrence of a specific event, and creates a story. It then references the user's DB, extracts material content according to the story structure, selects the material content to be used, and performs editing to generate moving content.

[0093] The decision to start automatic production based on event information can be made by detecting anticipated events, such as birthday parties, Mother's Day, and dates of events of interest, based on user attribute information such as birthday, sex, and preferences. Also, based on the user's schedule data, the occurrence of a particular event, such as a visit home, a trip, or a date, can be detected. In addition, it is possible to determine the occurrence of an event based on the user's behavior using SNS information related to the user. Furthermore, by analyzing images provided as material content, it is possible to determine events, such as athletic meets.

[0094] Workflow WF6 is an example of automatic production of moving content based on event information, but shows a case where material content is insufficient. The user's schedule data and SNS information are checked, a specific event is detected, and a story is conceived. Next, the user's DB is referenced and material content is extracted according to the story structure, but if there is insufficient material content, images of the missing scenes are collected by instructing the user to shoot or by performing some kind of automatic shooting as a material content supplement process. After that, when the necessary material content is collected, editing process is performed to generate moving content.

[0095] The workflow WF7 shows another example of a material content supplement process when material content is insufficient, similar to the workflow WF4 (or WF6). In cases where material content is insufficient for a set story, instead of instructing new shooting, it is possible to collect it from other databases, etc., or to create the necessary images. For example, material content can be searched for by referencing the target user's family database, friends database, public database, etc., or images of the necessary scenes can be generated as image synthesis / fake images, etc.

[0096] A workflow WF8 is an example in which a process for feeding back a user's evaluation at the time of viewing is added to the workflow WF3, for example. When the moving content is actually delivered as a work, or when the user is allowed to preview it before the final delivery, the user can perform some sort of evaluation operation. Feedback information according to the operation is reflected in the automatic content production process, such as story conception, selection of material content, and editing (scene order, etc.). FIG. 5 shows that the learning unit 17 learns about the feedback information dFB, and the learning result is reflected in the processing of the story generation unit 16. This may be reflected in the revision of the moving content, or the user's preferences may be learned and reflected in the production of future moving content. In other words, this is an example of using feedback from the user to customize the story concept, content selection, and editing method for the user.

[0097] The above workflows are merely examples, and various other examples are conceivable. Of course, it is also conceivable that different workflows can be used depending on the situation or the user.

[0098] Incidentally, it has been explained that workflow WF8 provides feedback on user evaluations when viewing moving content, but it is also possible to provide feedback on evaluations and relationships of material content. As described above, the processing of the evaluation unit 15 sets a relevance evaluation value and an individual evaluation value for each material content, but it is also possible for the evaluation unit 15 to set relevance presentation information for each individual material content to present to the user the relevance with other material contents and the evaluation value.

[0099] 7, 8, 9, and 10 show examples of display contents based on the relevance presentation information. These are one display mode for showing the relevance and evaluation value of material contents (CT1, CT2, . . . ) to a user when the material contents are registered in a DB for a certain user.

[0100] In Fig. 7, material contents CT1, CT2, etc. are displayed first. In the figure, each material content is indicated by "CT1" and "CT2" written within a circle, but in reality, thumbnail images showing the image content of each material content may be placed. Also, for example, a video may be played in response to mouseover or click. In any case, it is desirable to display the image content of each material content in a way that makes it easy for the user to recognize.

[0101] For each material content, the individual evaluation value is displayed so that it can be recognized, for example, as a numerical value on a scale of 1 to 10. For example, in the example of Fig. 7, the numerical value "10" is assigned to the material content CT1, the numerical value "7" is assigned to the material content CT2, etc., so that the individual evaluation value set for each material content can be recognized. Of course, the evaluation value is not limited to numerical values, and the size, shape, color, brightness, and position of the image expressing the material content may be used to recognize whether the individual evaluation value of each material content is high or low.

[0102] Furthermore, the relevance evaluation value between each of the material contents is shown by differences in the display mode such as the number of lines, the color of the lines, the thickness of the lines, the type of the lines, etc. It is considered that the stronger the relevance between the material contents, the stronger the connection is expressed. In the example of Figure 7, the fact that material contents CT1 and CT2 are highly related, that material contents CT4 and CT8 are relatively less related, and that material content CT9 is not related to other material contents is expressed by the number and type of lines indicating the connections, or the presence or absence of lines. Of course, the relevance evaluation value may be expressed by adding numerical values ​​to the lines indicating the perspective, direction, or connection of the layout positions of the material contents.

[0103] For example, the evaluation unit 15 generates related presentation information for displaying such an image, and provides it to the story generation unit 16 . The story generation unit 16 can provide such relationship presentation information to the user side and have the UI unit 22 present it to the user.

[0104] The user can use the UI unit 22 to check the display contents as shown in FIG. 7 and can also modify the evaluation by operating the UI unit 22. For example, the user can modify the individual evaluation value of material content that is an image of an important scene or a favorite image to a higher value, manipulate material content that is related but has been evaluated as having a low relevance to increase the relevance evaluation value, and cut lines that are not related but are evaluated as having relevance. Such operation information is fed back to the server device 10 as feedback information dFB shown in Fig. 5, so that the evaluation information by the evaluation unit 15 can be corrected. Such feedback information dFB is also input to the learning unit 17 for learning, and is reflected in a function for selecting material content by the story generation unit 16, for example.

[0105] In the example of Figure 7, an individual evaluation value and a relevance evaluation value are displayed for each material content, but it is also possible to display only the relevance based on the relevance evaluation value, or to present each individual evaluation value. That is, the relevance presentation information may be any information that can present to the user, such as evaluations and relevance of the material content. It is also desirable to form a user interface that allows the user to perform operations to provide feedback information dFB for all or part of the presented content.

[0106] While FIG. 7 above shows an example of the relationship between material contents, various modes of displaying the relationship are possible. FIG. 8 is an example of presenting key elements and showing the material content associated with them. In the case of Figure 8, "joy," "anger," "sorrow," and "happiness" are each key elements, and the example shows material contents CT1 and CT2 corresponding to "joy," material contents CT7 and CT8 corresponding to "anger," material content CT3 corresponding to "sorrow," and material contents CT4, CT5, and CT6 corresponding to "happiness."

[0107] 9 is also an example of presenting key elements, where a certain event (e.g., a piano recital) is set as the key element, and material contents CT1, CT2, CT3, CT4, and CT5 related to the event are presented. In this example, material contents CT6, CT7, and CT8 that do not directly correspond to the event but are related to material contents CT4 and CT5 related to the event are also presented.

[0108] Figure 10 shows an example of using a composite key element. As key elements, "family" and "friends" in the relationships between user A and "soccer" are presented, and user A's hobby "soccer" is also presented. This is an example of presenting material content related to these relationships and key elements, as well as the relationships between the material content.

[0109] As described above, various examples of the presentation of the relationship presentation information are possible, but it is desirable to make it easy for the user to recognize the relationship between each of the material contents. Of course, in the examples shown in Figs. 8, 9, and 10, the individual evaluation values ​​may be displayed as described in Fig. 7, and the relationship may be expressed in a different manner.

[0110] <4. Automatic production process including content completion> A specific example of the automatic content creation process performed in the server device 10, particularly by the function of the story creation unit 16, will be described.

[0111] FIG. 11 shows the automatic production process of moving content by an information processing device serving as the server device 10. In step S101, the server device 10 performs a production start determination in response to a trigger. This trigger is a start determination process in response to a request received from a user as described in the workflow WF3 in Fig. 6, or a start determination process in response to an event or the like as described in the workflow WF5.

[0112] In step S102, the server device 10 performs a story setting process for the moving content to be produced this time. A detailed example of the story setting process is shown in FIG.

[0113] In step S120 in FIG. 12, the server device 10 performs a process of assembling a story, such as that illustrated in FIG. 4, and also performs a process of selecting material content to be applied to each scene that constitutes the story. Constructing a story is the process of setting up a beginning, development, twist and conclusion for a certain theme of events, and constructing a setting such as chapters CH1 to CH5 in Figure 4. Also, one or more material contents to be used for each chapter are selected. For example, among the material contents stored in the DB corresponding to the user for whom moving content is to be produced, candidate contents that are likely to be applicable to the scenes of each chapter (for example, one chapter is assumed to be composed of one or more scenes) are extracted. In this case, for example, if the hobbies and preferences of the user and his / her family can be determined from the user information, it is effective to extract material contents with images that match the hobbies of the user. Then, the material content to be used for each scene is selected from the candidate contents. Of course, it is desirable for the user's preferences to be reflected in this selection as well. A more specific example of the process in step S120 will be described later with reference to FIG.

[0114] After the story is set and the corresponding material contents are selected in the process of step S120, the server device 10 determines in step S121 whether or not there is a shortage of material contents. If it is determined in step S120 that the story can be set and all the necessary material contents have been prepared, the process proceeds to step S124, where it is determined that content supplementation is unnecessary (for example, the supplement flag is turned off), and the process of Fig. 12 ends. That is, the process proceeds from step S102 to step S103 in Fig. 11.

[0115] In this case, since the complement flag is off, the server device 10 determines in step S103 that complement of the material content is not required, and proceeds to step S105 to perform editing processing. In this editing process, the server device 10 uses the chapter structure set in the story setting process and the selected material contents to generate moving content that makes up a series of stories. For example, the server device 10 cuts out necessary scenes from the videos as the selected material contents, adjusts the image quality, adds image effects, and then connects the material contents in accordance with the progress of the chapters.

[0116] When the impressive content has been completed through such editing processing, the server device 10 performs processing to provide the impressive content to the user side in step S106. For example, moving content such as movies may be made available to users for viewing by download or streaming. The above is the processing procedure for the cases illustrated by the workflows WF3 and WF5 in FIG.

[0117] On the other hand, there are cases where it is determined in step S121 of FIG. 12 that the material contents are insufficient. For example, there may be cases where material content applicable to a certain scene does not exist. Alternatively, there may be material content that is suitable for a certain scene, but it may be determined that no suitable material content exists because the evaluation value (individual evaluation value or relevance evaluation value) is low and the content is not suitable for use, or the image quality is low.

[0118] However, even if some of the material content is missing, for example, if deleting that scene does not significantly change the story structure, or if the scene is not particularly important, the lack of material content may be acceptable. Also, the user may intentionally input information that determines that the material content of that scene is unnecessary. In that case, the lack is also acceptable. In other words, in such cases, it is considered that allowing the lack, that is, intentionally not including a certain scene, is more in line with the user's intentions. If the server device 10 determines that the shortage of material contents is within an allowable range, the process proceeds from step S122 to step S124, and then to step S105 in FIG. 11 as described above.

[0119] However, if important material content is missing, or if logic is adopted that does not tolerate a lack of material content required for scene composition, the server device 10 proceeds to step S123 in FIG. 12, where it determines whether or not the story can be changed. For example, the server device 10 evaluates the importance of a scene lacking material content in the story, the possibility of a partial change to the story within a chapter, and the like, to determine whether or not a story change is possible. User preferences based on learning may also be added to the determination factors. For example, if the current user can evaluate that a certain scene is not important, the story change may be allowed.

[0120] If the story can be changed, the process returns from step S123 to step S120, and the story is constructed and the material contents are selected again.

[0121] If it is determined that there is a shortage of material content, that the shortage cannot be tolerated in the current story, and that a change to the story is also inappropriate, the server device 10 proceeds from step S123 to step S125, determines that material content supplement processing is necessary (for example, sets the supplement flag to ON), and ends the processing in Fig. 12. That is, step S102 in Fig. 11 is ended, and the processing proceeds to step S103. In this case, since the complement flag is on, the server device 10 determines in step S103 that material content complement is required, and proceeds to step S104 to execute material content complement processing. In other words, this is the situation described in the workflows WF4 and WF6 in FIG.

[0122] An example of the material content supplementation process performed in step S104 in FIG. 11 will be described. FIG. 13 shows an example of a material content supplement process in which a request is made to an external device to record material content that constitutes a story. In step S140, the server device 10 performs a process of requesting the external device to record the material content, that is, to record images and the like obtained by shooting. The external devices are the various devices shown in FIG. 1 and correspond to the material content provider 20 in FIG.

[0123] For example, the server device 10 transmits a request such as "Please take a video of the scene in question" to the mobile terminal 5 or the like, which is an external device, as a message to the user. It is also conceivable that a request message such as "Please take a video of Mr. A's ____ scene" may be sent to the mobile terminals 5 of the user's family and friends. Alternatively, the server device 10 may transmit control information to an application on the mobile terminal 5, the agent device 8, a robot, the surveillance camera 9, or the like, instructing the application to capture or provide an image of a target scene.

[0124] That is, the material content request process in step S140 is a process of requesting or instructing some external device to record material content of the required scene and transfer it to server device 10 (so that it can be used as material content for creating moving content).

[0125] After carrying out the request process in step S140, the server device 10 waits for input of material contents in step S141. While waiting for the input of material contents, the server device 10 checks for a timeout or input of material contents in step S142, and waits for the timing of a re-request in step S143. For example, a timeout may be set to one week after the request, and a request may be made again at a predetermined timing before the timeout is reached.

[0126] If it is time to make a request again before the timeout period expires, the server device 10 returns to step S140 and performs the material content request process again. In this case, the same request process as the previous one may be performed, or the external device to which the request is made may be changed, or the contents of the request may be changed.

[0127] When the material content is input in response to the request for the material content, or when the material content is not input even when the time-out period has elapsed, the server device 10 ends the material content supplementation process from step S142.

[0128] After performing the material content supplementation process as shown in FIG. 13 in step S104 in FIG. 11, the process returns to the story setting process in step S102. In this case, the process described in FIG. 12 is re-executed, and if the necessary material content is acquired, the server device 10 performs a process in step S120 to supply the newly acquired material content for the scenes that were lacking in material content. In addition, if the material content supplementation process times out, or if the input material content is not of sufficient quality, or if it is determined that the scene content is not appropriate, it may be necessary to start over from the beginning of the story construction.

[0129] For example, if the processing of step S120 in FIG. 12 is performed after going through the material content supplementation processing as shown in FIG. 13, and the process proceeds from step S121 or step S122 to step S124, the processing proceeds to the editing processing of step S105 as the processing of FIG. 11, and moving content is generated.

[0130] FIG. 14 shows another example of the material content complementing process. This is an example of expanding the scope of material content extraction before requesting material content from an external device.

[0131] In step S150, the server device 10 performs a process of searching for material content that can be applied to a scene for which material content is lacking, by referring to a DB associated with other people or a public DB that is suitable for the user who is the subject of current production, based on the user information. For example, basically, in step S120 of Fig. 12, material contents are extracted from a DB associated with the user who is the production target. Then, when the material contents are insufficient, the search for material contents is expanded to DBs related to other people and public DBs.

[0132] Based on the user information, the other person may be the user's family, friends, colleagues, etc. The DB associated with the other person is a DB that manages images captured by people who fall under the category of "other people" in the service. It is also possible to search for material content taking into account the user's hobbies and preferences, for example, the types of images that the user or his / her family prefers. For example, if the hobbies and preferences of the user or his / her family can be determined from the user information, images that match the user's hobbies or material content that is determined or estimated to be a scene or image content that the user likes are searched for.

[0133] In step S151, the server device 10 determines whether or not the currently required material content has been found from a database associated with another person or a public database. If found, the material content supplementation process ends and proceeds to step S102 in FIG.

[0134] If it is determined in step S151 of FIG. 14 that the necessary material content has not been found, the server device 10 proceeds to step S152 and performs a material content request process on the external device. Then, the process waits for input in step S153, input of material contents or timeout in step S154, and waits for a re-request in step S155. The process from step S152 to step S155 is basically the same as the process from step S140 to step S143 in FIG.

[0135] However, even in the material content request process in step S152, the request for material content may be made taking into consideration the hobbies and preferences of the user and his / her family, for example, the type of images the user likes. For example, if the hobbies and preferences of the user or his / her family can be determined from the user information, a request is made to capture image content that matches the user's hobbies, or scenes or image content that are determined or estimated to be liked by the user. For example, to a user who is determined or estimated to have a hobby of jogging and likes running, a recommendation message such as "Please take photos of the relay race" or "We recommend taking photos near the finish line" is sent to the user at a child's sports day. This makes it easier for users to obtain material content that suits their preferences.

[0136] As described above, by widening the range of DBs in which material contents are searched before executing material content request processing to an external device, the possibility of quickly satisfying material contents can be increased. By using images taken by others, it is expected that the range of content will be expanded, for example by inserting images with a field of view that the user did not intend into moving content.

[0137] FIG. 15 also shows another example of the material content complementing process. In step S160, server device 10 performs processing to search for material contents applicable to scenes lacking material contents by referring to DBs associated with other people who are suitable for the user of the current production target or public DBs based on the user information. This is the same processing as in step S150 in FIG. 14.

[0138] In step S161, the server device 10 determines whether or not the currently required material content has been found from a database associated with another person or a public database. If found, the material content supplementation process ends and proceeds to step S102 in FIG.

[0139] If it is determined that the necessary material content has not been found, the server device 10 proceeds from step S161 to step S162, and performs a process of acquiring a fake image and a cut-out image. The term "fake images" here refers to images that are not actual photographs of a subject, and refers to synthetic images, CG images, animated images, etc. The server device 10 generates images required for a scene by image synthesis processing, CG processing, etc. Alternatively, the server device 10 instructs an external device to execute such image generation, and acquires the generated images. Alternatively, the server device 10 may extract a part of an image from another material content to be used as material content adapted to the scene. For example, a scene in which the user is captured may be extracted from a security camera image or an image captured by another person.

[0140] By carrying out these processes, the server device 10 generates the missing material contents and uses them for the corresponding scene. After completing the process of step S162, the server device 10 ends the material content supplement process, and proceeds to step S102 in FIG.

[0141] The process of step S162 in FIG. 15 may be executed when a timeout occurs when a material content request process is performed on an external device in FIG. 13 or FIG.

[0142] <5. Pre-production recommendations / complementary processing> 11 to 15 have been described as processing that takes place after the start of the production of moving content, but it is also possible for the server side to take the opportunity to produce moving content and recommend image capture, etc. to the user. An example of processing in such a case will be described below.

[0143] FIG. 16 shows an example in which the server device 10 determines the timing of recommendation to the user based on the user information and performs the material content complementation process.

[0144] In step S201, the server device 10 determines the recommended timing for acquiring materials for moving content in accordance with an event that will occur in the future. For example, similar to the production triggers described in workflows WF5 and WF6 in Fig. 6, the system checks for events that could be opportunities to produce moving content based on event information, schedule information, etc. Then, it determines whether or not it is an opportunity to recommend taking images, etc., to be used in the production of such moving content. In addition, recommendation opportunities may be determined according to the life stage of the user, not just events. For example, by checking information such as age progression, changes in family structure, or marriage and childbirth, turning points in life stages are considered as opportunities to create moving content to commemorate those turning points, and timings close to those turning points are determined as recommendation opportunities.

[0145] For example, for a certain target user, the server device 10 determines that the recommendation time is a time before an event that can become material, such as one week before a child's athletic meet or three days before the target user's wedding.

[0146] If it is determined that it is time to make a recommendation, the server device 10 proceeds from step S202 to step S203, and performs a material content supplement process for creating moving content in the future. This material content supplementation process is basically a process of making a request to the user such as "I would like you to shoot a certain scene" at an event or the like in which the user is about to participate.

[0147] An example of the material content complementing process in step S203 is shown in FIG. In step S220, the server device 10 refers to the user information of the recommendation target, and searches for suitable content (material content or moving content) by referring to other people's or public DBs. For example, if the target user has a child's athletic meet next week, content about other people's athletic meets can be extracted. In this case, "other people" do not have to be related to the user, and it is good to refer to the databases of completely unrelated people broadly.

[0148] In step S221, the server device 10 analyzes highly rated scenes and image contents in the extracted content of others, and identifies moving elements. The evaluation can use, for example, feedback information dFB for moving content, evaluation values ​​of material content, or scenes, angles of view, etc. that are often used in a similar event (e.g., athletic meet) as highly rated scenes based on general standards. For example, if a target user has a child's sports day next week, the target user can refer to content of other people's sports days, identify highly rated scenes and image contents, and use these as moving elements.

[0149] In step S222, the server device 10 generates shooting recommendation data based on the analysis in step S221. For example, the shooting recommendation data is generated using text or image data as information including shooting techniques, advice, etc., according to highly rated scenes, angles, etc.

[0150] Then, in step S223, the server device 10 performs a material content request process. In this case, the process is to transmit the shooting recommendation data together with the recommendation content to the mobile terminal 5 of the user. For example, along with a recommendation message such as "Your child's sports day is next week. Would you like to create some content?", shooting recommendation data generated by referring to other people's content, such as shooting techniques for the sports day, is sent.

[0151] By making such recommendations prior to an event, it is expected that users will be able to provide more desirable material content more easily, and that the moving content they create thereafter will also be more satisfying. In particular, since it includes an element of being able to refer to how other people take pictures, it can provide useful information even to users who are not accustomed to taking pictures.

[0152] FIG. 18 shows an example of the process of step S203 in FIG. In step S230, the server device 10 performs a process of constructing a provisional story based on the event information for which a recommendation is to be made. Then, in step S231, the server device 10 sets what kind of material content is required as a scene according to the set provisional story, and generates shooting recommendation data indicating the details of such a scene.

[0153] Then, in step S232, the server device 10 performs a material content request process. In this case, the process is to transmit information presenting the contents of the scene as shooting recommendation data together with the recommendation contents to the user's mobile terminal 5 or the like. For example, along with a recommendation message such as "Your live performance will be next week. Would you like to create some content?", the app will suggest desirable scene content such as "scenery from the venue," "stage shots," "dressing room scenes," and "after-party scenes" depending on the story that is set.

[0154] By making such recommendations prior to an event, it will be easier for users to provide material content that matches the story, and as a result, it is expected that the moving content produced thereafter will also be highly satisfying.

[0155] <6. Evaluation and feedback of material content> Next, the evaluation and feedback process of material content in the server device 10 will be described. 5, the evaluation unit 15 sets a relevance evaluation value and an individual evaluation value for material content, and also performs a corresponding process, that is, an update process of the evaluation value, in response to feedback information dFB from the user.

[0156] One reason for setting and updating the evaluation values ​​in this way is to enable appropriate use in content production when considering that the server device 10 stores and manages a huge amount of content (memorable videos and photos) as a large number of material contents for a large number of users. For example, since each material content is fragmentary and does not have a storyline, there is a high possibility that it will be lost if it is simply stored. Therefore, by using AI processing to create a mining map of analysis results from different perspectives, it is possible to express the clarity of content with a strong story and the degree of connection based on importance and relevance as evaluation values. This makes it easier to select valuable material content when generating moving content, and aims to improve the quality of automatically produced moving content.

[0157] In addition to simply setting an evaluation value, the evaluation value will be visualized as relevance presentation information, enabling users to provide feedback. This will enable the relevance evaluation value and individual evaluation value to be revised to better suit individual users, and the evaluation function for creating moving content to be updated.

[0158] First, FIG. 19 shows a material content evaluation process performed by the server device 10 using the function of the evaluation unit 15.

[0159] In step S300, the server device 10 checks whether feedback information dFB relating to the material content has been input. If feedback information dFB has been input, feedback handling processing is carried out in step S320, as will be described later.

[0160] If the feedback information dFB has not been input, the server device 10 proceeds to step S301 to identify the material content to be processed. This means that material content that has been saved but has not been evaluated, such as newly uploaded material content, is identified as the target of the current evaluation process. However, material content for which an evaluation value has already been set may also be processed in order to be evaluated again depending on the passage of time, etc.

[0161] In step S302, the server device 10 calculates an individual evaluation value for the material content that is the subject of the evaluation process. As described above, the individual evaluation value is a value that evaluates the value of the image or scene of the material content itself alone, the importance of the content, etc. For this reason, the server device 10 performs image analysis and user information analysis, and calculates the individual evaluation value by a predetermined calculation using indices such as the relevance of emotions such as joy, anger, sadness, and happiness, whether the image was targeted for tracking (focus) when captured, whether the image content is a sudden event, whether the image has positive elements, whether the image has negative elements, and the number of times the image has been played as part of the user's viewing history. In addition, if the hobbies and preferences of the user or his / her family can be determined from the user information, it is desirable to reflect the hobbies and preferences of the user or his / her family in the evaluation value. For example, a high individual evaluation value is set for an image that matches the user's hobbies, or for an image that is determined or estimated to be a scene or image content that the user likes.

[0162] In step S303, the server device 10 sets key elements for the material content. For example, key elements such as a specific emotion, the type of person involved, and the event title are set. These can be set by referring to the analysis results of the image content, user information, metadata added to the image, and the like.

[0163] In step S304, the server device 10 extracts related material contents using the key elements, such as other material contents photographed at the same event as the material content to be evaluated, or other material contents photographed of the same specific person (e.g., a child) as the material content to be evaluated.

[0164] In step S305, the server device 10 calculates a relevance evaluation value for the material content to be evaluated. For example, the degree of relevance between material contents having a common key element is quantified to calculate a relevance evaluation value. The degree of relevance can be determined or estimated from image content such as the causal relationship of events, the chronological relationship, and the presence of people or objects related to the events. In addition, when the hobbies and preferences of the user and his / her family can be determined from the user information, it is desirable to reflect the hobbies and preferences of the user and his / her family in the evaluation value. For material contents that are causally related to each other, a high relevance evaluation value is set between material contents that are determined or estimated to be images related to the user's hobbies or to include scenes or image content that the user likes.

[0165] In step S306, the server device 10 generates or updates the relationship presentation information. This involves, for example, a process of generating new relationship presentation information as described with reference to Fig. 7 to Fig. 10, or a process of incorporating the material content currently being evaluated into existing relationship presentation information.

[0166] In step S307, the server device 10 stores the calculated individual evaluation value, relevance evaluation value, and key element for the material content to be processed, thereby completing the evaluation process.

[0167] When feedback information dFB relating to the material content is input, the server device 10 performs, for example, the process shown in FIG. 20 as the feedback handling process in step S320.

[0168] In step S321, the server device 10 identifies material content to be fed back. In step S322, the server device 10 determines whether or not the individual evaluation value of the material content identified as the feedback pair needs to be corrected based on the feedback information dFB. If necessary, the server device 10 proceeds to step S323 and performs an update process of the individual evaluation value.

[0169] In step S324, the server device 10 determines whether or not the individual evaluation value of the material content identified as the feedback target needs to be corrected based on the feedback information dFB. If necessary, the server device 10 proceeds to step S325 to update the key element set for that material content.

[0170] In step S326, the server device 10 determines whether or not the relevance evaluation value of the material content identified as the feedback target needs to be modified based on the feedback information dFB. If necessary, server device 10 proceeds to step S327 to reset related material contents for that material content, that is, other material contents whose relatedness is presented in the relatedness presentation information. In step S328, server device 10 calculates and updates the relevance evaluation value between the newly set related material content and other related material content. In step S329, the server device 10 updates the relationship presentation information in response to updates of other related material contents and the relationship evaluation values ​​between them.

[0171] By carrying out the process shown in FIG. 20, the evaluation value set by the evaluation unit 15 of the server device 10 is successively updated in accordance with the feedback information dFB from the user.

[0172] As explained above, the feedback information dFB from the user is reflected not only in the evaluation value of the material content, but also in the creation of moving content. 21 and 22, the process of step S120 in the story setting process previously described with reference to FIG. 12 will be described in detail, and an example in which the feedback information dFB is reflected will be described.

[0173] 12, a story is constructed and material contents are selected according to the story. Specifically, the server device 10 performs the process shown in FIG. 21, for example.

[0174] In step S401, the server device 10 creates a story. It is assumed that the story will be constructed as shown in the example in Figure 4. For example, the story will be constructed by arranging the components corresponding to the introduction, development, twist and conclusion. Also, for example, if a provisional story is constructed for a certain event in step S230 of Fig. 18, it is advisable to construct a story based on that, since it is assumed that material contents suitable for a scene of that event may have been provided by the user.

[0175] In step S402, the server device 10 collects candidate contents. That is, for each chapter constituting the story, one or more candidate contents to be used are collected from a DB of the user who is the subject of production. At this point, multiple material contents may be set as candidate contents to be used for one scene. For example, the candidate contents are collected based on the scene contents, user information, metadata, etc., regardless of the evaluation value.

[0176] In step S403, material content to be used for each scene constituting the story is selected. For example, a method of selecting material content with a high individual evaluation value from among many candidate contents for one scene may be used, but it is preferable to reflect relevance and the like from a more multifaceted perspective. For example, in addition to selecting material content with a high individual evaluation value for each scene, the relevance evaluation value between each of the selected material contents is also reflected, and multiple material contents are optimized as a selection result. The combination, order, and duration of the material contents are also included as selection elements. An evaluation function including elements such as the individual evaluation value, the relevance evaluation value, the duration, and the sequence is used to select material content to be used in each scene constituting the story.

[0177] Generally, material content with high individual evaluation values ​​and relevance evaluation values ​​is likely to be selected, but as these evaluation values ​​are updated in accordance with feedback information dFB based on user operations as described in Figures 19 and 20, material content that suits the user's preferences tends to be selected. Furthermore, when setting the individual evaluation values ​​and relevance evaluation values ​​in Figure 19, by setting evaluation values ​​that reflect the preferences of the user and his / her family members, etc. (what types of images and scenes they prefer, etc.) that can be determined or estimated from the user information, the material content applied to the scene will be more likely to reflect the user's preferences.

[0178] In step S404, the server device 10 generates edited data based on the result of the selection of the material content in step S403. The editing data includes information on the selected material contents, information on the order in which the material contents are joined, information on the processing of each material content (image quality adjustment, extraction, effects, etc.), etc. In other words, it is design information for creating moving content.

[0179] For example, such a process is performed as step S120 in FIG. 12, and later, editing process based on the editing information is executed in step S105 in FIG.

[0180] Here, when feedback information dFB based on a user operation is input, the processing of FIG. 22 can be performed by the functions of the learning unit 17 and the story generation unit 16.

[0181] If the feedback information dFB is input, the server device 10 advances the process of FIG. 22 from step S420 to step S421. If the feedback information dFB requests a change in the story of the moving content currently being produced, the process proceeds from step S421 to step S401 in Fig. 21, and the story is reconstructed. For example, it is assumed that some scenes will be cut or replaced based on the feedback information dFB. It is assumed that the feedback information dFB for moving content currently being produced is transmitted in response to a user operation, for example, when the server device 10 allows a user to preview the moving content before the content is actually delivered. If the feedback information dFB indicates that the current material contents need to be changed, the server device 10 proceeds from step S422 in FIG. 22 to step S403 in FIG. This causes the material content selection in step S403 to be redone.

[0182] The feedback information dFB is not necessarily reflected in the moving content being created. For example, it may be feedback information dFB in response to presentation by related presentation information. However, since user operations also affect the story structure, it is desirable to have the information learned for future story generation. For example, if the information is to be reflected in future content creation, the server device 10 proceeds from step S423 to step S424 and performs learning / updating processing of the evaluation function. For example, some of the coefficients and variable elements of the evaluation function used when creating moving content for that user are updated. This, for example, modifies the direction of material content selection and the elements to be emphasized in step S403 in Fig. 21. In other words, the material content selection process is adapted according to the preferences of that user.

[0183] In the above, we have mainly explained examples of adapting the generated moving content to the user by generating an evaluation function and performing learning / update processing as the functions of the evaluation unit 15 and the learning unit 17. However, instead of the above processing, it is also possible to use a learning model learned using machine learning and deep learning so that the generated moving content is adapted to the user. For example, the story generation unit 16 may include a function substituting for the evaluation unit 15, and generate moving content while evaluating, classifying and extracting material content. Specifically, the story generation unit 16 performs deep learning based on various information about the user and generates a learning model that suits the user. A learning model is then applied to each piece of material content to classify and extract it, and to determine whether or not the material content can be applied to the story. After that, material content determined to be relevant is used to generate moving content.

[0184] In this way, the evaluation unit and story generation unit 16 are provided with a function for generating a learning model and a function for generating contents using the learning model. Furthermore, the learning section 17 performs re-learning based on the user feedback information dFB and updates the learning model, which is suitable for generating more desirable moving content.

[0185] <7. Management of negative / positive content> Some material content can be classified as negative or positive content. It is assumed that negative content is usually not something that users want to watch. However, as explained in Figure 4, when considering that negative content is structured as a story for personal content, it can be a useful material content for enhancing the emotion when watching. On the other hand, for example, if the server device 10 functions as cloud storage and provides personal storage space for users in the storage 10a, negative content may be viewed as easily as other material content by users, and may be considered undesirable.

[0186] Therefore, in the server device 10, the function of the storage management section 18 is such that the negative contents are stored and managed separately from other material contents. For example, a remote storage 12 is prepared in which accessibility from users is not ensured, or which is less accessible than the storage 10a even from the server device 10, and negative contents are stored therein.

[0187] FIG. 23 shows a material content management process performed by the server device 10 using the function of the storage management unit 18. In step S501, the server device 10 identifies the material content to be processed.

[0188] In step S502, the server device 10 determines whether the material content to be processed is negative content or not. Whether or not an image is negative content may be determined by some kind of processing. For example, if the image includes metadata about the user's vital signs at the time of shooting, it may be possible to determine whether or not negative emotions are occurring based on the vital signs, or it may be possible to estimate negative emotions from the user's facial expressions, actions, and gestures through image analysis. A user's failure to view a page may be considered a negative factor and added to the evaluation factors. Also, characteristics that differ from other everyday images may be considered negative elements and added to the judgment factors. In addition, if there is an interval between everyday images, it can be assumed that some negative event has occurred, and this can be added to the inferred factors, as images from such intervals are likely to contain negative elements.

[0189] It should be noted that positive content can also be determined using a method similar to that for determining negative content.

[0190] If it is determined in step S502 that the content is not negative, the server device 10 proceeds to step S503 and performs normal storage of the material content. For example, the material content is stored in the storage 10a so as to be managed in a DB associated with the user.

[0191] If it is determined in step S502 that the content is negative, the server device 10 proceeds to step S504 and performs a storage process for the material content corresponding to negative content. For example, the material content is stored in the remote storage 12 so as to be managed in a DB associated with the user.

[0192] In this way, negative content is stored separately from other material content. Although the negative content-compatible storage in step S504 is exemplified by the storage in the remote storage 12, the remote storage 12 may be any of the following storages.

[0193] A storage device that is separate from the storage 10a and located in a remote location A storage device that is less accessible to the server device 10 than the storage device 10a -Storage devices that are managed to restrict user access -Storage devices accessible via communication paths with low transfer rates Storage devices connected to standalone appliances The storage 10a is connected to the server device 10 via a first network, whereas the remote storage 12 is a storage device connected to the server device via a second network having a communication speed slower than that of the first network.

[0194] The above is just one example, but it is possible to assume that the remote storage 12 is not normally accessed.

[0195] Furthermore, the process of step S504 is not necessarily limited to recording in the remote storage 12. For example, negative content may be stored in the storage 10a in the same way as other material content. However, accessibility may be differentiated by performing hierarchical management to make access difficult, encrypting the content, or setting access permission only for authorized persons.

[0196] The above process in FIG. 23 is an example in which negative content is stored and managed separately from other material content, but negative content, positive content, and other material content may be managed separately. In this case, the negative contents are made less accessible by separating the storage, but the positive contents and other material contents may be made equally accessible. For example, the positive contents and other material contents are stored in the same way in the DB of the corresponding user in the storage 10a, but in the case of the positive contents, a flag indicating that it is positive content may be set. In addition, all material contents may be divided into negative and positive contents. For example, contents that do not fall under the category of negative contents may be managed as positive contents.

[0197] As described above, negative content is managed separately. However, since negative content can be useful when used as material for creating moving content, it is desirable to change the management status according to the situation. Therefore, it is possible to carry out the processing shown in FIG.

[0198] The sequential storage process in step S520 indicates a process in which the server device 10 sequentially stores material contents, for example, in the process shown in FIG. 23, in response to the uploading of the material contents, for example.

[0199] Here, in step S521, the server device 10 checks whether the currently stored material content is positive content or not. If the content is not positive, that is, if the content is negative or is material content that is neither negative nor positive, the processing in FIG. 24 ends.

[0200] If the content is positive, the server device 10 advances the process to step S522, and searches for negative content related to the current positive content. For example, if the current positive content is about the user doing well in a baseball game, processing is performed to search for negative content of a scene in which the user was injured during practice in the past.

[0201] If the corresponding negative content is not found, the process ends from step S523, but if the corresponding negative content is found, the server device 10 performs a process to change the management status of the negative content in step S524. For example, a process to improve accessibility for the server device 10 is performed, such as moving the storage device from the remote storage 12 to the storage 10a.

[0202] The server device 10 may not enter the negative content managed as the negative content-corresponding storage in step S504 of FIG. 23 as a candidate content in the process of the story generation unit 16. In other words, negative content is not unconditionally used in creating moving content. However, in the process of step S524, if it is determined that corresponding positive content exists, the management status of the negative content may be changed so that it is eligible to be considered as candidate content. In other words, it is only when the negative content has acquired importance in the story of highlighting the positive content that it can be used in creating moving content.

[0203] In step S525, the server device 10 updates the evaluation value of the negative content in response to the occurrence of a relationship between the negative content and the currently stored positive content, i.e., recalculates and updates the relationship evaluation value and the individual evaluation value.

[0204] By the above process, it is possible to create or increase the possibility that negative content corresponding to positive content will be applied to a certain scene in the production of moving content. This is expected to enhance the story quality of moving content that is automatically produced using appropriate negative content.

[0205] FIG. 25 shows a similar processing example. The sequential storage process in step S530 indicates a process in which the server device 10 sequentially stores material contents, for example, in the process shown in FIG. 23, in response to the uploading of the material contents, for example.

[0206] Here, in step S531, the server device 10 checks whether the currently stored material content is negative content or not. If the content is negative, a process for registering a negative event is performed in step S532. For example, information about what negative event occurred and when for a certain user is registered as one piece of user information.

[0207] If it is determined in step S531 that the material content is not negative content, the server device 10 determines in step S533 whether the currently stored material content is positive content corresponding to a negative event previously registered. If the material content saved this time is positive content related to a negative event previously registered in the processing of step S532, the server device 10 proceeds to step S534 and performs processing to change the management status of the negative content related to the registered negative event. For example, a process is performed to improve accessibility for the server device 10, such as moving a storage device from the remote storage 12 to the storage 10a. In this case, the server device 10 may also change the state of the negative content to a management state so that the content is eligible to be a candidate content.

[0208] In step S535, the server device 10 updates the evaluation value of the negative content in response to the occurrence of a relationship between the negative content and the currently stored positive content, i.e., recalculates and updates the relationship evaluation value and the individual evaluation value.

[0209] The above process also creates or increases the possibility that negative content corresponding to positive content will be applied to a certain scene in the production of moving content.

[0210] <8. Summary and variations> According to the above embodiment, the following effects can be obtained. A server device 10 as an information processing device according to the embodiment includes a story generating section 16 that performs processing to request an external device to record material content that constitutes a story for moving content to be automatically produced (see FIGS. 11 to 18). This process is useful for collecting moving content material that features the customer's personal actions, experiences, etc. in the story. In particular, by making a specific request such as what kind of scene you would like to have filmed, this process is suitable for collecting material content.

[0211] In the embodiment, an example is given in which the story generation unit 16 sets the story of the moving content, selects the material content to be used, and performs a story setting process to determine whether there is a shortage of material content, and if it determines that there is a shortage of material content, performs a process to request an external device to record the material content (see Figures 11, 12, 13, and 14). By collecting material content according to a story and requesting recording from an external device if there is a shortage, material content supplement processing can be executed efficiently in the production of automatically produced content.

[0212] In the embodiment, an example has been given in which the story generation unit 16 performs a recommendation time determination process for a user who is considered to be a customer, and performs a process of requesting an external device to record material content in response to the recommendation time determination process (see FIG. 16). That is, the opportunity for automatic content production for a user who is a customer of the automatic content production service is determined based on the user's event schedule, life stage, etc. Then, a process is performed to request the recording of material content according to the determined time. By making recommendations to users (customers) on things like taking pictures to record, or requesting recording on various devices, in response to upcoming events or changes in their lives, it becomes easier to obtain material content for opportunities that contribute to the memories and emotions of the users. By using such material content, it is possible to increase the possibility of providing automatically produced content that satisfies the users. The period relating to a user who is considered a customer includes not only the user himself, but also periods based on various events and life stages relating to his family and friends.

[0213] In the embodiment, an example was given in which the story generation unit 16 determines the user's taste based on the user information, and requests the recording of material content based on the determined taste information. For example, taste determination is performed based on user information on the user himself / herself, his / her family and friends who are considered to be customers, and a request to record material content is made based on this (see FIG. 14). By making a recording request based on user information about the user himself / herself, his / her family, relatives, friends, etc. who is considered to be a customer, it becomes easier to collect material content that matches the preferences of the viewer user, and it is possible to increase the possibility of realizing automatically produced content that matches the preferences.

[0214] In the embodiment, an example was given in which the story generation unit 16 determines the user's taste based on the user information, and selects material content using the determined taste information as a selection factor. For example, tastes are determined or estimated based on user information on the user himself / herself, his / her family and friends, who are considered to be customers. An evaluation value reflecting such tastes is set, and a process of selecting material content using the taste information as a selection factor is performed (see step S403 in FIG. 19 and FIG. 21). By using user information about the user himself / herself, his / her family, relatives, friends, etc., who is considered to be a customer, it is possible to select material content that matches the preferences of the user on the viewer side according to the story. This increases the possibility of realizing automatically produced content that matches the preferences of the user.

[0215] In the embodiment, an example has been given in which the story generation unit 16 performs processing to request the recording of material content based on event information related to a user (see Figs. 16 and 18). For example, an appropriate scene or the like according to the type of event information of a user who is considered to be a customer is determined, and a request is made to record the appropriate material content. There are many types of events, such as sports matches, recitals, weddings, children's athletic meets, musical performances, and theatrical performances, and each event can have its own desirable scenes. By requesting the shooting of such scenes, the possibility of realizing a more complete automatically produced content can be increased.

[0216] In the embodiment, the story generating section 16 may perform a process of selecting material content based on event information related to the user. If a story corresponding to an event is tentatively constructed in step S230 in Fig. 18 and a story is constructed based on that in step S201 in Fig. 21, the material contents selected in step S403 will be selected based on the event information. In other words, material contents that suit appropriate scenes, etc. according to the type of event information of the user who is considered to be a customer are selected. This increases the possibility of realizing highly complete automatically produced content suited to the event.

[0217] In the embodiment, an example of recommendation timing determination processing is given in which timing determination is made based on the life stage of the user (see FIG. 16). For example, timings such as birth, admission to school, graduation, employment, marriage, birth of children, child growth, retirement, etc. are determined as the life stage of the user, and appropriate material content is selected and a recording request is made accordingly. The progression and changes of a user's life stages have a narrative quality, making them suitable for creating automatically produced content. Therefore, by determining the timing according to the life stage, it is possible to request shooting, etc. at a timing suitable for collecting suitable material content for automatically produced content. This makes it easier to collect material content desirable for automatic production.

[0218] In the embodiment, an example has been given in which the story generation unit 16 generates recommendation data indicating the details of the material content requested to be recorded, based on the event that caused the timing determination process to determine that it is time to request recording, and content of other users regarding the same event (see Figure 17). For example, the event that is the cause of the determination that the time to request recording in the timing determination process is an event in the user's life stage (admission, graduation, employment, marriage, etc.), various events, etc. When it is determined that the time to request recording is due to such an event, data is generated that recommends the contents of the scene to be photographed, etc., based on the content of the same event for other people. The server device 10 generates shooting recommendation data indicating recommended content for a user's event, life stage, etc., by referring to other people's content (material content or automatically produced content) at the same event or life stage. The server device 10 then transmits the data to the user's terminal device and presents it to the user. This allows the user to know what kind of shooting is recommended for the current event, making it easier to obtain material content suitable for automatically produced content and to realize automatically produced content that satisfies the user.

[0219] In the embodiment, the story generation unit 16 performs a story setting process for automatically producing content for a user who is a customer, selects material content to be used from a DB corresponding to the user, and determines whether the material content is insufficient. If it is determined that there is a shortage of material content, the story generation unit 16 searches for material content from other DBs (e.g., DBs corresponding to other users or public DBs), and if it is determined that there is a shortage of material content, performs a process of requesting an external device to record the material content (see FIG. 14). This allows a wide range of material content to be searched for, including not only still images and videos uploaded by customer users, but also images taken by other people, such as close friends and family, complete strangers, and even public images. This allows content to be automatically produced using a wider variety of image stocks. Also, by minimizing recording requests, it is possible to increase the possibility of content being produced quickly and without hassle for the user. In other words, the efficiency of material content supplementation processing can be improved.

[0220] The server device 10 as an information processing device of the embodiment includes an evaluation unit 15 that performs a process of setting an evaluation value for material content recorded in an accessible state, and a story generation unit 16 that selects material content that constitutes the story set in the story setting process by referring to the evaluation value, and automatically creates content by editing processing using the selected material content (see Figures 19 to 22). By having material content selected with reference to evaluation values ​​(individual evaluation values ​​and relevance evaluation values) during the production process of automatically produced content, it is possible to increase the likelihood of using material content that is more desirable to the user, thereby making it possible to provide automatically produced content that satisfies the user.

[0221] In the embodiment, an example has been given in which story generation unit 16 performs a process of collecting material content that is to be used as a candidate according to the set story, and selects the material content to be used for automatic content production from the material content collected as candidates using evaluation values ​​(see FIG. 21). The evaluation value given to each piece of material content makes it possible to easily and appropriately select material content suitable for the current automatic production from among the candidate contents. For example, candidate contents corresponding to a certain scene can be collected based on the scene contents, but it is difficult to select the most suitable material content for the current automatically produced content or the material content that will impress the current user from among them based on the image contents alone. Therefore, the selection is made taking into consideration the evaluation values ​​that reflect the relationships between the material contents and the importance of the material contents themselves. This is believed to enable more desirable content production.

[0222] In the embodiment, an example has been given in which evaluation section 15 sets a relevance evaluation value indicating the relevance of a material content with other material content (see FIG. 19). For example, in the case where a material content has a causal relationship or other relevance with other material content, an evaluation value indicating the relevance with the other material content is set. By evaluating each piece of material content not just in isolation, but in relation to other content, it is believed that it will be possible to appropriately match the contents used within a single automatically-produced piece of content, making it possible to automatically produce more moving content.

[0223] In the embodiment, an example has been given in which the evaluation section 15 sets an individual evaluation value for each piece of material content (see FIG. 19). Although there is a huge amount of content for each material content, the importance of each content varies. The importance varies depending on the general scene quality, the scene that expresses the user's emotions, and so on. Furthermore, determining the importance of each candidate content during automatic production imposes a large processing load. Therefore, an evaluation value for the content itself is set in advance, and selection can be made using that evaluation value. This makes it possible to easily and appropriately select material content to be used in automatically produced content, which is believed to enable the automatic production of more moving content.

[0224] In the embodiment, an example has been given in which the evaluation section 15 sets, for a material content, relationship presentation information that presents the relationship between the material content and other material content (see FIG. 19 and FIGS. 7 to 10). For example, as shown in Figures 7 to 10, by generating relationship presentation information showing the relationship between each piece of material content, it is possible to present the user with the existence of material content and its relationships. This makes it easier for the user to recognize the types of material content and its deficiencies.

[0225] In the embodiment, an example has been given in which the relationship presentation information and the relationship evaluation value are updated in response to a user operation (see FIG. 20). For example, by presenting the relationship presentation information in the form shown in Figures 7 to 10, the user can check whether the relationship between the material contents is appropriate. In addition, it is possible to perform operations such as lowering the relationship between material contents that are actually less related, and the relationship information can be corrected by performing update processing according to the feedback information dFB. This makes it possible to converge the relationship evaluation value and the relationship presentation information to those that are more appropriate for the user. The individual evaluation values ​​are also updated in accordance with the feedback information dFB, so that they converge to a state that reflects the user's hobbies and preferences.

[0226] The relationship presentation information in the embodiment shows the relationship between the material contents by directly linking one material content with another material content. For example, the relevance between the contents is presented on the user interface by using an image that links the material contents together (see Figs. 7 to 10). This allows the user to easily check whether the relevance between the material contents is appropriate.

[0227] In the embodiment, the relevance presentation information shows an example of the relevance of material contents in a form in which material contents are linked to key elements (see Fig. 8 and Fig. 9). For example, a word or event showing an emotion is used as a key element, and the relevance of content is presented on the user interface in a form such as material contents corresponding to a certain emotion, material contents related to a certain event, and so on. By showing the relationships between material contents using the emotions of joy, anger, sadness, and happiness as key elements as in Figure 8, or by providing an image showing the relationships between material contents using a certain event as a key element as in Figure 9, users can easily confirm the relevance of material contents to their own emotions, events, etc., and their appropriateness. Other possible key elements include life stages, specific keywords, types of activities, time periods, seasons, locations, and user ages.

[0228] In the embodiment, an example has been given in which the relationship presentation information indicates the relationship of the material content in a form in which the material content is linked to the user's personal relationships (see FIG. 10). For example, as shown in Fig. 10, material contents related to family members, material contents related to friends, etc. are presented to user A. This allows the user to easily check the relevance of material contents to their personal relationships and their suitability. Other possible interpersonal elements that present related material content include, from the perspective of user A, parents, children, members of a certain club, classmates, coworkers, friends with similar hobbies, neighbors, and so on.

[0229] The server device 10 of the embodiment includes the learning unit 17 that inputs a user operation on relevance presentation information and performs learning processing. The learning unit 17 performs a learning process to reflect the feedback information dFB from the user in response to the relevance presentation information in the story setting. This enables future story setting that reflects the user's intention, the importance and relevance of the material content, etc., and enables story correction of the automatically produced content currently being produced.

[0230] In the embodiment, an example has been given in which the learning contents by the learning unit 17 are reflected in the story correction. When proceeding from step S421 in Fig. 22 to step S401 in Fig. 21, the story of the automatically produced content currently being produced is corrected based on the learning results. This increases the possibility that content suitable for the user can be produced.

[0231] In the embodiment, an example has been given in which the learning contents by the learning section 17 are reflected in the selection of material content. When proceeding from step S422 in Fig. 22 to step S403 in Fig. 21, the material content to be adopted is reselected according to the story of the moving content currently being produced based on the learning results. This increases the possibility that the user will be able to produce automatically produced content using material content that is more suitable for the user.

[0232] In the embodiment, an example has been given in which the learning contents by the learning unit 17 are reflected in the automatic production of future content. For example, in steps S324 and S329 of Fig. 20, the individual evaluation value and the relevance evaluation value are updated. Also, in step S424 of Fig. 22, the evaluation function is updated so that the learning content is reflected in future automatic production. This allows the learning content to be reflected in future automatic content production. The automatic production system for moving content can be evolved so that it can produce automatically produced content that is more desirable for users.

[0233] The server device 10 as an information processing device according to the embodiment includes a memory management unit 18 that stores negative content determined to be a negative scene among material contents constituting the story of automatically produced content in a state in which the negative content can be distinguished from other material contents (see FIGS. 23 to 25). When creating moving content, material content is selected according to the set scene, but by also appropriately selecting negative content and arranging it in the story structure, it is believed that more moving content can be generated. By storing and managing negative content separately from other material content, it is possible to easily and appropriately select negative content when creating such automatically created content. In the embodiment, an example has been described in which negative content and positive content are distinguished and managed, but negative content may be determined, and the negative content may be managed separately from other material content.

[0234] In the embodiment, an example has been given in which the memory management unit 18 performs processing for storing material content by distinguishing between negative content and positive content determined to have a positive scene detail (see FIGS. 23 to 25). It is desirable that positive and negative contents are appropriately selected in scenes set during the production process of moving content. By storing and managing the negative and positive contents separately, it is possible to easily and appropriately select the negative and positive contents when producing such automatically produced content.

[0235] In the embodiment, an example is given in which the memory management unit 18 stores material content determined to be negative content in a remote storage 12 (second storage device) different from the storage 10a (first storage device) that stores and manages material content other than negative content. By storing negative content and other content (positive content) in separate storage devices, it is possible to properly distinguish between the content and negative content, and to make it difficult for users to see the content. For example, it is assumed that material content managed by a cloud server can be used for automatically produced content, and can also be viewed simply as videos or still images saved by users. However, negative content is usually not something users want to look back on, and in some cases, they may want to forget about it. In addition, the images may be taken by someone other than the user (such as a robot). In such cases, the user may not want to have the images seen by themselves or others. Therefore, it is possible to store the images in a separate storage and manage them in a way that they are not usually viewed.

[0236] In the embodiment, it is assumed that storage 10a is a storage device that is managed in a state where user access to the material content is permitted, and remote storage 12 is a storage device that is managed in a state where only specific administrators are permitted to access the material content. For example, the storage 10a and the remote storage 12 have different access permissions. In this way, negative content can be generally hidden from the user.

[0237] In the embodiment, an example has been described in which the remote storage 12 is a storage device whose access speed from the storage management unit 18 is slower than that of the storage 10a. For example, the storage 10a and the remote storage 12 have different access speeds due to differences in network interior and exterior, communication methods, etc. In this way, negative content can be made less visible to general users. In order to slow down the communication speed, the storage 10a may be, for example, a storage device in the server device 10, or a storage device connected to a LAN, or a storage device connected to a normally used network, while the remote storage 12 may be a storage device connected to a network that performs communication at a low transfer rate, a storage device located outside the earth, such as in a desert area outside an urban area or on the moon, or a storage device connected to a stand-alone computer. In any case, a storage device that is inconvenient for normal browsing can be assumed.

[0238] In the embodiment, an example is assumed in which the storage 10a is accessible by the memory management unit 18 via a first network, and the remote storage 12 is accessible by the memory management unit 18 via a second network. Network differentiation allows accessibility to negative content to be different from other material content.

[0239] In the embodiment, an example has been described in which the storage management unit 18 stores material content determined to be negative content and material content other than negative content in the storage device in different hierarchical management states. For example, negative content and other material content are managed in folders with different hierarchical levels using a storage device of a server that manages material content. Or, even if different storage devices are used, the hierarchical management status is made different. Hierarchical management can also make negative content less accessible than other content (positive content).

[0240] In the embodiment, an example has been given in which the memory management unit 18 performs processing to facilitate access by the memory management unit 18 when it determines that negative content is related to positive content (see Figures 24 and 25). For example, negative content is normally stored and managed in a state where accessibility is reduced, but in response to confirmation of the existence of related positive content, the content is stored and managed in a state where accessibility is improved. Negative content that has related positive content is considered to have the potential to be selected as one of the scenes that make up moving content, making it easier for the server device 10 to access and easier to provide as candidate content when producing automatically produced content. This makes it possible to make negative content that is normally managed in a state that is difficult to view easier to use in moving content.

[0241] In the embodiment, it has been described that the memory management unit 18 treats negative content as material content that can be candidate content for use in the automatic content production process in response to determining that the negative content is related to positive content. In other words, negative content is not normally considered as a candidate for use in automatically produced content, but it can be considered as a candidate content in response to confirmation of the presence of related positive content, thereby making it possible to prevent negative content from being used in moving content without due consideration. In addition, by making it possible to use negative content in automatic content creation only when there is positive content that corresponds to it, the content will not simply be a negative scene, but will be used as content for a story that leads to a positive event afterwards. This allows negative content to be used effectively.

[0242] In the embodiment, an example of automatically producing moving content such as a movie using material content mainly as moving images has been described, but it is of course possible to use still image data, audio data, text data, etc. as material content, or to use these data in combination. As an example of automatically produced content, a customer user's personal movie has been described as "moving content," but this is not necessarily limited to such personal moving content, and content of various kinds, such as content about groups or organizations, documentary video, academic video, etc., can be broadly considered as "automatically produced content" in the present disclosure.

[0243] The program of the embodiment is a program that causes, for example, a CPU, a DSP, or a device including these to execute the processes described in the embodiment.

[0244] The program according to the embodiment is a program that causes an information processing device to execute a process of requesting an external device to record material content that constitutes a story for automatically produced content. In addition, the program of the embodiment is a program that causes an information processing device to execute a process of setting an evaluation value for material content recorded in an accessible state, and a story generation process of selecting material content that constitutes the story set in the story setting process by referring to the evaluation value, and automatically producing content by editing processing using the selected material content. In addition, the program of the embodiment is a program that causes an information processing device to execute a memory management process that stores negative content determined to be negative scene content among the material content that makes up the story of the automatically produced content in a state where it can be distinguished from other material content.

[0245] By using such a program, the above-mentioned server device 10 can be realized in a computer device or other device capable of executing information processing.

[0246] A program for implementing such a server device 10 can be recorded in advance in a HDD as a recording medium built into a device such as a computer device, or in a ROM in a microcomputer having a CPU. Alternatively, the software may be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media may be provided as a so-called package software. Furthermore, such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0247] Furthermore, such a program is suitable for a wide range of applications of the server device 10 of the embodiment. For example, by downloading the program to a mobile terminal device such as a smartphone or tablet, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), or the like, the smartphone or the like can function as the server device 10 of the present disclosure.

[0248] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0249] The present technology can also be configured as follows. (1) A story generation unit is provided for requesting an external device to record material content that constitutes a story for automatically produced content. Information processing device. (2) The story generation unit performs a story setting process for setting a story of the automatically produced content, selecting material content to be used, and determining whether the material content is insufficient, and performs a process for requesting an external device to record the material content when it is determined that the material content is insufficient. The information processing device according to (1) above. (3) The story generation unit performs a timing determination process for a user who is a customer, and performs a process of requesting an external device to record material content in response to the timing determination process. The information processing device according to (1) or (2) above. (4) The story generation unit determines a user's preference based on the user information, and performs a process of requesting recording of material content based on the determined preference information. An information processing device according to any one of (1) to (3) above. (5) The story generation unit determines a user's preference based on the user information, and performs a process of selecting material content based on the determined preference information. The information processing device according to (2) above. (6) The story generation unit performs a process of requesting recording of material content based on event information related to a user. An information processing device according to any one of (1) to (5) above. (7) The story generation unit performs a process of selecting material content based on event information related to a user. An information processing device according to any one of (2) and (5) above. (8) The timing determination process is performed based on the user's life stage. The information processing device according to (3) above. (9) The story generation unit includes: Generate recommendation data indicating the details of the material content to be requested to be recorded based on the event that caused the determination in the timing determination process that the time for requesting recording is to be made and the content of other users regarding the same event. The information processing device according to (3) or (6) above. (10) The story generation unit includes: A story setting process is performed for a user who is a customer, in which a story of the content to be automatically produced is set, material content to be used is selected from a database corresponding to the user, and a story setting process is performed to determine whether or not there is a shortage of material content; In response to a determination that the material contents are insufficient, the material contents are searched for in another database, and if it is further determined that the material contents are insufficient, a process is performed to request an external device to record the material contents. The information processing device according to (1) above. (11) An information processing device, For automatically produced content, a process is performed to request an external device to record the material content that makes up the story Information processing methods. (12) For automatically produced content, a process that requests an external device to record the material content that makes up the story A program to be executed by an information processing device. [Explanation of symbols]

[0250] 2 Camera 3. Computer equipment 4. Tablet devices 5. Mobile devices 6 Camera-equipped aircraft 7 Camera-equipped vehicles 8 Agent Device 9. Surveillance Cameras 10. Server equipment 10a Storage 11 Network 12 Remote Storage 15 Evaluation Section 16 Story Generation Unit 17 Learning Department 18 Memory Management Department 20 Material Contents Department 21 User Information Department 22 UI section

Claims

1. a story generation unit that performs a process of requesting an external device to record material content that constitutes a story for automatically produced content; The story generation unit includes: A timing determination process for a user who is a customer; a process of requesting an external device to record material content in response to the time determination process; and generating recommendation data indicating details of material content for which recording is requested based on the event that is the cause of the determination that the time for recording is to be requested in the time determination process and content of other users regarding the same event. Information processing device.

2. The story generation unit performs a story setting process for setting a story of the automatically produced content, selecting material content to be used, and determining whether the material content is insufficient, and performs a process for requesting an external device to record the material content when it is determined that the material content is insufficient. The information processing device according to claim 1 .

3. The story generation unit determines a user's preference based on the user information, and performs a process of requesting recording of material content based on the determined preference information. The information processing device according to claim 1 .

4. The story generation unit determines a user's preference based on the user information, and performs a process of selecting material content based on the determined preference information. The information processing device according to claim 2 .

5. The story generation unit performs a process of requesting recording of material content based on event information related to a user. The information processing device according to claim 1 .

6. The story generation unit performs a process of selecting material content based on event information related to a user. The information processing device according to claim 2 .

7. The timing determination process is performed based on the user's life stage. The information processing device according to claim 1 .

8. The story generation unit includes: A story setting process is performed for a user who is a customer, in which a story of the content to be automatically produced is set, material content to be used is selected from a database corresponding to the user, and a story setting process is performed to determine whether or not there is a shortage of material content; In response to a determination that the material contents are insufficient, the material contents are searched for in another database, and if it is further determined that the material contents are insufficient, a process is performed to request an external device to record the material contents. The information processing device according to claim 1 .

9. An information processing device, A timing determination process for a user who is a customer; A process of requesting an external device to record material content constituting a story for the content to be automatically produced in response to the time determination process; and generating recommendation data indicating details of material content for which recording is requested based on the event that is the cause of the determination that the time for recording is to be requested in the time determination process and content of other users regarding the same event. Information processing methods.

10. A timing determination process for a user who is a customer; A process of requesting an external device to record material content constituting a story for the content to be automatically produced in response to the time determination process; A process of generating recommendation data indicating details of material content to be requested to be recorded, based on the event that is the cause of the determination that the time for requesting recording is the time in the time determination process and content of other users regarding the same event; A program to be executed by an information processing device.

Images