Information processing apparatus, information processing method, program

Abstract

This information processing device is provided with a storage management unit which saves a negative content determined as being a negative scene content, among material contents constituting a content story to be automatically generated, in a manner as to be discriminable from the other material contents.

Description

Technical Field

[0001] This technology relates to an information processing apparatus, an information processing method, and a program, and particularly relates to the technical field of automatic production of content using images and sounds.

Background Art

[0002] The following Patent Document 1 discloses a technique for producing video content by connecting unit videos that are materials.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, due to the miniaturization of recording devices such as imaging devices and recording devices, their mounting on portable devices, or the spread of various devices, etc., imaging of images (images are a general term for videos and still images) and recording of sounds have become easier and more widespread. In addition, due to the diversification of recording media such as memory cards and the spread of cloud storage using network communication, etc., the storage modes of data such as photos, videos, and sounds have also become diversified. In such an environment, for example, it has become relatively easy for a server device to acquire images of customer users, etc., and as a service, it is conceivable to produce content such as a short movie starring customer users. Furthermore, due to the progress of AI processing, etc. in recent years, it is also conceivable to accurately collect images, etc. that are materials and automatically produce content desired by customer users.

[0005] Here, considering that the server device providing the service performs automatic content production and provides it to customers, it is assumed that there may be cases where negative scenes are required for the user in terms of story composition. On the other hand, pixel materials and the like that are negative scenes for the user may not be what the user himself / herself wants to watch or save. Therefore, the purpose of this technology is to enable the appropriate management of material content with negative content for the user and to be able to use it for automatic production.

Means for Solving the Problem

[0006] The information processing apparatus according to this technology includes a storage management unit that stores negative content determined to be negative scene content among the material content that constitutes the story of the content to be automatically produced in a state where it can be distinguished from other material content.

[0007] Consider automatically producing content with a certain story using a plurality of material contents. Material content is an image (video or still image), voice, etc. that constitutes each scene that makes up the story. Among this material content, negative content is distinguished. For example, material content determined to have scene content such as pessimistic, negative, or regressive is regarded as negative content. Such negative content is stored and managed separately from other material content.

[0008] In the information processing apparatus according to the above-described technology, it is conceivable that the storage management unit performs a process of storing the material content by distinguishing between the negative content and positive content determined to have positive scene content. Material content is classified into positive content and negative content. For example, material content determined to have scene content such as optimistic, affirmative, or progressive is regarded as positive content. These positive content and negative content are stored and managed separately.

[0009] In the information processing apparatus according to the present technology described above, it is conceivable that the memory management unit stores the material content determined to be the negative content in a second storage device different from the first storage device that stores and manages the material content other than the negative content. For example, negative content is stored using a storage device different from the storage device of the server that manages material content.

[0010] In the information processing apparatus according to the present technology described above, it is conceivable that the first storage device is a storage device managed in a state where user access to the material content is permitted, and the second storage device is a storage device managed in a state where access to the material content is permitted only to a specific administrator. For example, the first storage device and the second storage device shall have different access rights.

[0011] In the information processing apparatus according to the present technology described above, it is conceivable that the second storage device is a storage device whose access speed from the memory management unit is slower than that of the first storage device. For example, the first storage device and the second storage device shall have different access speeds due to differences such as inside and outside the network and communication methods.

[0012] In the information processing apparatus according to the present technology described above, it is conceivable that the first storage device is accessible by the memory management unit via a first network, and the second storage device is accessible by the memory management unit via a second network. For example, the first storage device and the second storage device shall be accessed via different networks.

[0013] In the information processing apparatus according to the present technology described above, it is conceivable that the memory management unit stores the material content determined to be the negative content and the material content other than the negative content in a storage device in different hierarchical management states.

[0014] For example, using the storage device of a server that manages material content, the negative content and other material content are managed in folders of different hierarchies. Or even when using different storages, the hierarchical management states are made different.

[0015] In the information processing apparatus according to the present technology described above, it is conceivable that, in response to the determination that the memory management unit is related to positive content determined for the negative content, the memory management unit performs a process to facilitate access by the memory management unit.

[0016] Normally, negative content is stored and managed in a state where its accessibility is reduced, but in response to confirming the existence of related positive content, it is managed in a state where its accessibility is improved.

[0017] In the information processing apparatus according to the present technology described above, it is conceivable that, in response to the determination that the memory management unit is related to positive content determined for the negative content, the memory management unit makes the negative content into material content that can be a candidate content for use in content automatic production processing. That is, normally, negative content is not a candidate for use in automatic production content.

[0018] The information processing method according to the present technology is an information processing apparatus in which the information processing apparatus executes a memory management process for storing, in a distinguishable state from other material content, the negative content determined to be negative scene content among the material content constituting the story of the content to be automatically produced. Thereby, the negative content is appropriately managed.

[0019] The program according to this technology is a program that causes an information processing apparatus to execute such an information processing method. This facilitates the realization of the information processing apparatus of this technology.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Embodiment for Carrying Out the Invention

[0021] Hereinafter, the embodiments will be described in the following order. <1. System Configuration> <2. Overview of Automatically Produced Content> <3. Flow of Content Production> <4. Automatic Production Process Including Content Completion> <5. Recommendation / Completion Process Before Production> <6. Evaluation and Feedback of Material Content> <7. Management of Negative / Positive Content> <8. Summary and Variation Examples>

[0022] In addition, in the present disclosure, the following terms are used to distinguish "content". · Automatically Produced Content In the present disclosure, for example, content provided to a user as a customer is automatically produced by an information processing apparatus. The content that is automatically produced and will be viewed by the user is referred to as "automatically produced content". However, in the embodiments, for the purpose of giving an example of automatically producing for the purpose of impressing customers such as users of the "automatically produced content", for convenience of explanation, it is mainly referred to as "impressive content". "Impressive content" is a specific example of "automatically produced content". As forms of automatically produced content, there are various types, including content including video / audio, content such as a slide show using still images, content including still images and audio, content consisting of only audio, content including video, still images, and audio, content including text data, and so on.

[0023] · Material content Automatically produced content is produced by combining a plurality of video materials, still image materials, audio materials, text materials, etc. The content data such as video, still image, audio, text, etc. serving as each material is referred to as "material content". Any content can be material content. · Candidate content Among the material contents, it refers to the material content that is regarded as a candidate to be used as an image or audio for a certain scene when producing automatically produced content. For example, among a large number of material contents, several candidate contents that seem to be applicable to a certain scene are extracted, and among the candidate contents, the material content to be used for the said scene is selected.

[0024] · Negative content and positive content Among the material contents, those determined to have scene contents such as optimistic, affirmative, progressive, etc. are regarded as "positive content". Among the material contents, those determined to have scene contents such as pessimistic, negative, regressive, etc. are regarded as "negative content". However, this "positive content" and "negative content" are not absolute classifications. Depending on the scene content type, or the customer user, or the time / era, or common sense, social situation, etc., some judgment criteria may be determined for judgment. In this technology, at least the material content determined to be "negative content" is managed separately from other material content.

[0025] Also, in this disclosure, "image" may refer to either a video or a still image, and is a term used as a general term for them.

[0026] <1. System Configuration> Fig. 1 shows an example of a system configuration including a server device 10. In the embodiment, an example will be described in which an information processing device constituting the server device 10 automatically creates moving content for customer users using a large number of video materials, etc. of a large number of users, and provides the moving content to the users. Also, the material content will be described by taking an example mainly of video data (including audio). That is, it is assumed that the automatically created moving content is like a movie created by editing a plurality of videos.

[0027] The server device 10 in Fig. 1 is an information processing device capable of communicating with various devices via a network 11 such as the Internet, and functions as, for example, a so-called cloud server.

[0028] The server device 10 can store a large amount of material content and created moving content in its storage 10a under its management. Note that the storage 10a represents a storage accessible by the server device 10, and may be a storage device installed inside the information processing device as the server device 10, or may be a storage device connected via a LAN (Local Area Network) or the like. Also, a storage located at a location away from the server device 10 or a user-side storage device may be used as the storage 10a here. That is, if the server device 10 is a storage that can collect material content for automatic generation of moving content, it may correspond to the storage 10a shown here.

[0029] Also, the server device 10 can store material content and the like in the remote storage 12. This remote storage 12 is shown as a storage device with reduced accessibility compared to the storage 10a. The remote storage 12 may also be a storage device within the server device 10 or an external storage device. Detailed examples will be described later. The remote storage 12 may be a storage device that cannot be physically accessed or accessed through management of access rights, etc. from user-side devices such as the illustrated computer device 3, tablet terminal 4, mobile terminal 5, etc.

[0030] Such a server device 10 may be a server that provides a cloud storage service on a network, or a server that provides an SNS (Social Networking Service), or a platform server that provides a comprehensive service.

[0031] The network 11 is assumed to be, for example, the Internet, a home network, a LAN, a satellite communication network, or other various networks. Various devices can upload image data and the like to the server device 10 via the network 11, or download various information from the server device 10. Further, as part or all of the network 11, for example, communication by FTP (File Transfer Protocol) may be adopted. Also, short-range wireless communications such as Bluetooth (registered trademark), WI-FI (Wireless Fidelity: registered trademark), NFC (Near Field Communication: registered trademark), or infrared communication may be adopted. In any case, the network 11 can be assumed to be any network capable of data communication.

[0032] In FIG. 1, as devices capable of performing shooting and recording on the user side, a camera 2, a computer device 3, a tablet terminal 4, a mobile terminal 5, a camera-equipped aircraft 6, a camera-equipped vehicle 7, an agent device 8, and a surveillance camera 9 are shown. These are merely examples of devices capable of performing shooting and recording, and there can be other devices as well.

[0033] The camera 2 is an imaging device that can capture still images and moving images and record or transfer and output them as image data. It is assumed that the camera 2 is used by a large number of users, and the captured image data (moving images and still images) is stored in the storage 10a or the like, enabling the server device 10 to use those image data as material content.

[0034] The computer device 3 exemplifies, for example, a notebook personal computer, a desktop personal computer, etc. The tablet terminal 4 and the mobile terminal 5 such as a smartphone can also be said to be a form of the computer device. In the case of these computer devices 3, tablet terminals 4, mobile terminals 5, etc., for example, image data captured by a built-in camera, image data transferred from other devices such as the camera 2, image data generated as composite images or computer graphics (CG), image data as video chat images, image data shared with family and friends, etc. are stored in the storage 10a or the like, and it is assumed that the server device 10 can access them.

[0035] The camera-equipped flying object 6 is assumed to be a flying object equipped with a camera such as a drone. It is assumed that the image data obtained by shooting with the camera-equipped flying object 6 is stored in the storage 10a or the like and can be accessed by the server device 10. It is assumed that drones and the like are owned by users, are public property, or are owned by operators of emotional content providing services and are used to shoot customer users for material content collection.

[0036] The camera-equipped vehicle 7 is a vehicle such as a general automobile, and is a vehicle equipped with a camera such as a drive recorder. For example, by providing the image of the drive recorder of the vehicle owned by the user to the server device 10, it is assumed that the image data obtained by shooting with the camera-equipped vehicle 7 is stored in the storage 10a or the like and can be accessed by the server device 10.

[0037] The agent device 8 is an interface device for providing services such as information provision and home appliance control in, for example, a home, and it is also assumed that it can be photographed by mounting a camera. It is assumed that the image data obtained by shooting with the agent device 8 is stored in the storage 10a or the like and can be accessed by the server device 10. Although exemplified as the agent device 8 here, for example, a housework support robot or a pet-type robot used at home also has a photographing function, and it is assumed that the image data obtained by their shooting is stored in the storage 10a or the like and can be accessed by the server device 10.

[0038] The surveillance camera 9 includes, for example, those installed in public places or those installed in the private space by general users. It is assumed that the image data obtained by shooting with such a surveillance camera 9 is stored in the storage 10a or the like and can be accessed by the server device 10.

[0039] The various devices as described above can be devices that can obtain the material content used by the server device 10, or can be devices with a user interface.

[0040] Fig. 2 illustrates the relationship between the server device 10 and the various devices. The various devices shown in Fig. 1 may function as the material content providing unit 20 in Fig. 2, or may function as the UI (user interface) unit 22.

[0041] For example, the above-described devices such as the camera 2, the computer device 3, and the surveillance camera 9 can capture or generate content such as video data, and thus can be a source of the material content that can be used by the server device 10. That is, the images obtained by the various devices can be the material content for creating moving content as the personal movie of the customer user. This is merely an example, and although there are other considerations besides the following, examples of the images that can be the material content are as follows.

[0042] · Images taken and recorded by the customer user himself / herself and uploaded to the cloud · Images captured by a vendor upon a service request · Images provided by the release of the customer user's local data · Images taken and uploaded by the customer user's friends, family, etc. · Images taken by robots / drones / in-vehicle cameras, etc. · Images of area cameras, surveillance cameras, etc. · Images on the SNS of oneself / friends / followers, etc. · Images during communication such as video chats · Images created by image processing such as CG images and fake images · Public images and image images that are permitted for free use

[0043] These images are stored, for example, in the storage 10a or the remote storage 12 in a state where they can be managed as images corresponding to individual users. When a certain user A becomes a customer and creates moving content, the images associated with that user A become usable material content. Each image data may be collectively stored in association with the identification information of the individual user, or multiple user identification information may be assigned to one image so that one image can be used for multiple users.

[0044] Also, the camera 2, the computer device 3, the mobile terminal 5, etc. are devices equipped with UI functions such as display and operation input. Therefore, when these devices display moving content and various images provided by the server device 10 to the user, or when receiving user operations, they can function as the UI unit 22.

[0045] Also, FIG. 2 shows an example of the functional configuration of the server device 10. As functions related to the automatic production process of moving content, the server device 10 includes an evaluation unit 15, a story generation unit 16, a learning unit 17, and a memory management unit 18.

[0046] The evaluation unit 15 is a processing function that sets an evaluation value for each material content stored in, for example, the storage 10a and accessible by the server device 10. The story generation unit 16 is a function that performs the story setting process of moving content for customer users, the selection process of material content according to it, the editing process of the selected material content, etc., and automatically produces moving content. The learning unit 17 is a function that performs learning processing adapted to individual users by receiving feedback from the users. The memory management unit 18 performs memory management of material content, particularly memory management of negative content and positive content. Specific processing examples of these functions will be described later, but by performing, for example, AI processing as each function, high-quality processing is realized.

[0047] Also, FIG. 2 shows the function as the user information providing unit 21. This user information providing unit 21 may be a function built into devices such as the camera 2, the computer device 3, the tablet terminal 4, and the mobile terminal 5, or it may be a separate device or sensor device. It may also be a wearable device capable of acquiring user information (such as position information indicating the user's location and vital information) at each point in time.

[0048] And the user information includes, for example, various information such as attribute information such as the user's age and gender, hobbies, the user's face, family composition, the ages of family members, information about friends, etc., schedule information, event information (schedules, participation history, SNS information, etc.), the user's camera history, vital information, and position information.

[0049] Some of these user information are associated with and managed for the individual user, and some are associated with material contents such as image data. For example, there is some that is added to and associated with image data as metadata. For example, attribute information such as the user's age and gender, hobbies, the user's face, family composition, the ages of family members, information about friends, etc., schedule information, event information, and camera history may be associated with the individual user. The server device 10 manages such user information for each user as information for appropriately creating moving contents for the user. Also, in the sense of information at the time of shooting, information such as the user's age at the time of shooting, family composition at the time of shooting, information about friends, etc., schedule information and event information corresponding to the shooting content (information about what event the image is from), vital information at the time of shooting, and position information at the time of shooting can be regarded as user information associated with the image. These user information can be stored as metadata for each material content, for example, or managed separately as an information file corresponding to the material content. Thereby, the server device 10 can use the corresponding user information for each material content.

[0050] Also, the user information provided by the user information providing unit 21 may include not only directly detected information, but also information obtained by, for example, the result of image analysis or the analysis of a plurality of sensing data. For example, through the analysis of captured image data, there is also information such as the characteristics of the user's appearance, expression, complexion, and the emotions inferred from them. Also, when vital sensors measure things like heart rate, brain sensing information, blood pressure values, and sweating amount, the user's emotions can be inferred from them and used as user information.

[0051] Also, information about the human relationship at the time of shooting is useful. For example, information such as the type of image taken by the customer user being processed himself / herself, the image taken by a friend of the customer user, the image taken by a family member of the customer user, and the image objectively taken by a surveillance camera can also become user information associated with the image.

[0052] As described above, these user information obtained by the user information providing unit 21 are provided to the device that becomes the material content providing unit 20 and added to the image data as metadata, or provided to the server device 10 and managed in association with the user individual or the material content.

[0053] Next, FIG. 3 shows an example of the hardware configuration of the information processing device 70 that can be configured as the server device 10, the computer device 3, the tablet terminal 4, the mobile terminal 5, and the like.

[0054] The CPU (Central Processing Unit) 71 of the information processing device 70 executes various processes according to the programs stored in the ROM (Read Only Memory) 72 or the programs loaded from the storage unit 79 to the RAM (Random Access Memory) 73. The RAM 73 also appropriately stores the data and the like necessary for the CPU 71 to execute various processes. The CPU 71, ROM 72, and RAM 73 are interconnected via a bus 74. An input / output interface 75 is also connected to this bus 74.

[0055] An input unit 76 composed of an operator and an operation device is connected to the input / output interface 75. For example, as the input unit 76, various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, and a remote controller are assumed. The operation of the user is detected by the input unit 76, and the signal corresponding to the input operation is interpreted by the CPU 71.

[0056] In addition, to the input / output interface 75, a display unit 77 composed of a liquid crystal panel (LCD: Liquid Crystal Display), an organic EL (Electro-Luminescence) display, etc., and an audio output unit 78 composed of a speaker, etc. are connected integrally or separately.

[0057] The display unit 77 is a display unit that performs various displays, and is composed of, for example, a display device provided on the housing of the information processing apparatus 70, a separate display device connected to the information processing apparatus 70, etc. The display unit 77 executes displays of images for various image processes, moving images of processing targets, etc. on the display screen based on instructions from the CPU 71. Also, the display unit 77 performs displays as various operation menus, icons, messages, etc., that is, as a GUI (Graphical User Interface) based on instructions from the CPU 71.

[0058] To the input / output interface 75, a storage unit 79 composed of a hard disk, a solid-state memory, etc., and a communication unit 80 that performs communication using various communication methods may be connected.

[0059] The communication unit 80 performs communication processing via a transmission path such as the Internet, wired / wireless communication with various devices, bus communication, etc. When this information processing apparatus 70 is considered as the server apparatus 10, the communication unit 80 at least has a function of performing communication via the network 11.

[0060] The input / output interface 75 is also connected with a drive 82 as required, and a removable recording medium 81 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted thereon. The drive 82 can read data files such as image files and various computer programs from the removable recording medium 81. The read data files are stored in the storage unit 79, or the images and sounds included in the data files are output by the display unit 77 and the audio output unit 78. Also, computer programs and the like read from the removable recording medium 81 are installed in the storage unit 79 as required.

[0061] In this information processing apparatus 70, for example, software for the processing of the present disclosure can be installed via network communication by the communication unit 80 or via the removable recording medium 81. Alternatively, the software may be stored in the ROM 72, the storage unit 79, or the like in advance.

[0062] When assuming the server apparatus 10 as the information processing apparatus 70, in the information processing apparatus 70, the functional configuration described in FIG. 2 is constructed in the CPU 71 by software (application program).

[0063] <2. Outline of Automatically Produced Content> The server apparatus 10 creates moving content as a personal movie for a certain customer user as automatically produced content. The outline of the story configuration of this moving content will be described.

[0064] FIG. 4 shows an example of the story configuration of moving content and its corresponding variations. Here, as an example of the story configuration, an example is given that includes the rising action, climax, falling action, and conclusion in terms of content and has the following chapters CH1, CH2, CH3, CH4, CH5.

[0065] Chapter CH1: Past Daily Life Here, scenes showing what the user used to do every day are arranged. For example, baseball practice, piano practice, commuting to work, friendships, etc.

[0066] Chapter CH2: Incident Occurrence Arrange scenes showing events such as the reason or change that caused the user to stop (or become unable to do) what they used to do every day. For example, events such as an injury, a new friendship, a job transfer, graduation, etc.

[0067] Chapter CH3: Digest until Daily Life Returns For example, arrange negative content. Scenes showing the troubles during rehabilitation from an injury, bad habits, difficult things, lonely days, etc. are applicable.

[0068] Chapter CH4: Current Daily Life Arrange scenes showing the current daily life that has escaped from a negative state. For example, the return to baseball practice, dedication to piano practice, another job transfer, reunion with friends, etc.

[0069] Chapter CH5: Main Scene Arrange scenes that liven up the story (moving scenes). For example, scenes of returning to the game and excelling, a piano recital, achievements at work, the development of friendships, etc.

[0070] The above is just an example, but it is conceivable to generate personal moving content for customer users by featuring moving scenes while including the beginning, middle, transition, and conclusion in this way.

[0071] <3. Process of Content Creation> For example, the process of automatically creating moving content having such a story structure will be described. FIG. 5 shows an overview of the processing flows of the evaluation unit 15, the learning unit 17, and the story generation unit 16 as functions of the server device 10. The relationship with the material content providing unit 20, the user information providing unit 21, and the UI unit 22 is also shown.

[0072] As described above, for the server device 10, images, voices, and metadata (including user information associated with the material content) that can be material content are provided from the material content providing unit 20. In addition, user information is provided from the user information providing unit 21.

[0073] These material contents and user information are associated and stored and managed for each user individually by the function of the storage management unit 18 shown in FIG. 2 in the server device 10. For the stored material content, an evaluation value setting process by the evaluation unit 15 will be performed.

[0074] The evaluation unit 15 sets an evaluation value indicating the relevance between material contents for each material content, and also sets an individual evaluation value for each individual material content.

[0075] The relevance evaluation value for a certain material content is a value indicating the relevance between that material content and other material contents, and is, for example, a value obtained by quantifying the degree of relevance such as a causal relationship or a time-series relationship.

[0076] The individual evaluation value is a value for evaluating the value of the image itself, the value of the scene, the importance of the content, etc. for each individual material content. For example, if the image content expresses any of the emotions of joy, anger, sorrow, or pleasure, a high evaluation value is set. Also, if the subject is photographed as a tracking target, it is given a high evaluation value as material content for the user of that subject. Also, for images of sudden events or events not common in daily life, etc., a high evaluation value is given as material content useful in story composition. An image with positive elements or an image with negative elements is high - valued material content useful in story composition as positive content or negative content, respectively. Information on the viewing history of a user, such as whether the user liked it and watched it many times or never played it at all, represents the importance and favorites for the individual user. Therefore, an evaluation value can be set according to the viewing history.

[0077] The evaluation unit 15 sets, for each user who is a customer, the relevance evaluation value and the individual evaluation value for each material content associated with that user, for example, as described above. These settings can be made by comprehensively reflecting the determination result of the scene content by image analysis, the image quality, user information, etc.

[0078] Such evaluation values are provided to the story generation unit 16. Based on some trigger, the story generation unit 16 performs a process of generating personal touching content for a certain user. For example, the story generation unit 16 sets a story with the customer user himself or his family as the main character, and collects material content that seems usable as each scene constituting the story as candidate content. Then, for each candidate content, an operation using the evaluation value (evaluation function operation) is performed, and the material content actually used as touching content is selected. Then, an editing process is performed using the selected material content to generate touching content such as a movie with the customer user or his family as the main character.

[0079] The touching content produced in this way is provided to the user side and can be viewed by the user through the UI unit 22.

[0080] The above is the basic process of automatic content production. An example of the timeline of this automatic production is shown in FIG. 6. The time - series procedure is as follows. · Step ST1: Shooting videos, still images, etc. as material content · Step ST2: Transfer of material content from the user side to the server device 10 (made available for use by the server device 10) · Step ST3: Saving and management of material content on the server device 10 side · Step ST10: Conception of the story of the moving content · Step ST20: Selection of material content according to the set story · Step ST30: Post-processing such as image quality processing and scene extraction for the material content, and editing such as joining together the material content that has undergone such processing · Step ST40: Providing the moving content to the user side (viewing on the user side)

[0081] Here, steps ST10, ST20, and ST30 are the processes of automatic content production by the story generation unit 16.

[0082] This figure 6 shows various workflow examples. Workflows WF1 and WF2 show up to the collection of material content.

[0083] Workflow WF1 is an example in which, for example, video shooting using a camera 2 or the like is performed by a user, the video data is transferred to the server device 10, and a database (hereinafter, the database is denoted as "DB") for the user is generated or recorded in the DB, so that the material content for the user can be used.

[0084] Workflow WF2 is an example in which images for a certain user are collected by a method other than shooting by that user, and are managed in the DB for that user in the server device 10, so that they can be used as material content for that user. In this case, it is assumed that the image collection is either taken by other users or collected as public data. This is an example of collecting material content in a wider range, not just limited to the material content captured by the user themselves.

[0085] Workflows WF3 to WF8 show examples of automatic content production by the story generation unit 16.

[0086] Workflow WF3 starts with a request from the user to create touching content or the situation of predetermined material content as a trigger. First, conceive a story for a certain user, then refer to the user's DB and extract material content according to the story composition. Then select the material content to be used and perform editing processing to generate touching content.

[0087] As a trigger for automatic production of touching content, a production request from the user is assumed. For example, information indicating what kind of content is desired, such as specifying the theme, title, or specific material content of the touching content from the user, and based on this, the server device 10 automatically generates the touching content. For example, it is also conceivable that the user only specifies a part of the material content to be used in the touching content, and accordingly the server device 10 sets the story, selects the material content of the front and back scenes, and produces the touching content.

[0088] In addition, as a trigger for automatic production of touching content, not only the above-mentioned active instructions from the user side, but also, for example, a message from the server device 10 to the user saying "How about creating content like ○○?" and starting production when there is a corresponding request is also conceivable. For example, based on the collection status of material content, if material content related to the sports meet of the user's child has been collected, it is also conceivable to send a message such as "How about making a movie of your child's sports meet?" Also, when there is certain negative content and positive content corresponding to it can be obtained, in such a situation, a story as shown in FIG. 4 above can be constructed. Therefore, in such a situation, it is also conceivable to recommend content production to the user and start automatic production according to the user's response. Also, when there is material content that the user has forgotten, such as an old - dated photographed image or an image that the user has hardly viewed, it is also conceivable to recommend content production according to the content of the image to the user and start automatic production according to the user's response. Similarly, when there are images when the user starts a new activity, images taken at a place visited for the first time, or images taken at a place visited after a long time, it is also conceivable to recommend content production according to the content of the image to the user and start automatic production according to the user's response.

[0089] Note that the sending of these messages can be done by email, SNS messages, etc. to users registered with the service, or by display on the application screen of the terminal device (computer device 3, tablet terminal 4, mobile terminal 5, etc.) of users who have installed the application for service provision, push notifications, etc.

[0090] Furthermore, as a trigger for automatic production of touching content, it is also conceivable to automatically set a theme or title from the content of the material content, for example, and start automatic production of touching content without waiting for the user's request.

[0091] Workflow WF4 is an example where, in response to a trigger such as a request from a user, it conceives a story for a certain user, and then refers to the user's DB to extract material content according to the composition of the story, but there is a case where the material content is insufficient. In such a case, as a material content complementing process, by giving a shooting instruction to the user or performing some kind of automatic shooting, etc., images of the lacking scenes are collected. After that, when all the necessary material content is gathered, an editing process is carried out to generate moving content. The shooting instruction to the user can be notified, for example, by a message to the user, such as "Please take pictures of scenes like ○○". Also, the instruction for automatic shooting can be to instruct a robot or household device owned by the user to take pictures of a predetermined scene, or to instruct the equipment of a photography company.

[0092] Workflow WF5 is an example of automatically producing moving content based on event information. It checks the user's attribute information, schedule data, SNS information, etc., detects that a specific event has occurred, and conceives a story. Then it refers to the user's DB, extracts material content according to the composition of the story, selects the material content to be used, and performs an editing process to generate moving content.

[0093] The determination of the start of automatic production based on event information is possible, for example, by detecting assumed events such as birthdays, Mother's Day, the opening days of events of interest, etc. from the user's attribute information such as birthday, gender, hobbies, etc. Also, based on the user's schedule data, the occurrence of specific events, such as events like going back to one's hometown, traveling, dating, etc., can be detected. Also, based on the SNS information about the user, it is possible to determine the occurrence of an event based on the user's actions. Furthermore, event determination, such as determination of a sports meet, etc., is possible by image analysis of the images provided as material content.

[0094] Workflow WF6 is an example of automatically creating moving content based on event information, showing the case where there is a shortage of material content. It checks the user's schedule data, SNS information, etc., detects the occurrence of a specific event, and conceives a story. Next, it refers to the user's DB and extracts material content according to the story structure. When there is insufficient material content, as a material content supplementation process, it collects images of the missing scenes by giving the user a shooting instruction or performing some kind of automatic shooting. After that, when all the necessary material content is available, it performs an editing process to generate moving content.

[0095] Workflow WF7 shows another example of the material content supplementation process when there is insufficient material content, similar to Workflow WF4 (or WF6). When there is a shortage of material content according to the set story, instead of instructing new shooting, it is an example of collecting from other DBs, etc. or creating the necessary images. For example, it searches for material content by referring to the DB of the target user's family, friends, public DB, etc., or generates images of the necessary scenes as image synthesis / fake pictures.

[0096] Workflow WF8 is an example where, for example, a process of feeding back the user's evaluation during viewing is added to Workflow WF3. Regarding the viewing of moving content, when it is actually delivered as a work, or when the user is temporarily allowed to preview it before its final delivery, the user is enabled to perform some kind of evaluation operation. The feedback information corresponding to the operation is reflected in the automatic content creation processes such as story conception, selection of material content, and editing (scene order, etc.). Figure 5 shows that the learning unit 17 learns about the feedback information dFB and reflects the learning result in the processing of the story generation unit 16. This may be reflected in the operation of modifying the current moving content, or the preferences of the user may be learned and reflected in the production of subsequent moving content. That is, it is an example in which the story concept, material content selection, and editing method are specialized for the user by using feedback from the user.

[0097] The above workflows are just examples, and many other examples are assumed. Of course, it is also conceivable that the workflows are used appropriately according to the situation or the user.

[0098] By the way, as for workflow WF8, although it was explained that user evaluation when viewing moving content is fed back, it is also conceivable to feed back the evaluation and relevance of material content. As described above, in the process of the evaluation unit 15, a relevance evaluation value and an individual evaluation value are set for each material content. Furthermore, it is conceivable that the evaluation unit 15 sets relevance presentation information for presenting the relevance and evaluation value of individual material content to the user with other material content.

[0099] Examples of display contents by relevance presentation information are shown in FIGS. 7, 8, 9, and 10. These are a display mode when showing the relevance and evaluation value to the user for material contents (CT1, CT2...) registered in the DB for a certain user as material contents.

[0100] In FIG. 7, first, material contents CT1, CT2... are displayed. In the figure, "CT1" and "CT2" are shown in the circle to indicate each material content, but actually, it is conceivable to arrange thumbnail images showing the image content of each material content. Also, for example, the video may be played in response to mouse over or click. In any case, it is desirable to make the display such that the user can easily recognize the image content of each material content.

[0101] For each material content, it is displayed so that an individual evaluation value can be recognized, for example, with a 10 - level numerical value. For example, in the example of FIG. 7, numerical values such as "10" for material content CT1 and "7" for material content CT2 are assigned, and the individual evaluation values set for each material content can be recognized. Of course, not limited to numerical values, the individual evaluation values of each material content may be recognizable by differences in the size, form, color, brightness, and arrangement position of the images representing the material contents.

[0102] Also, between each material content, a relevance evaluation value is shown by differences in display modes such as the number of lines, line color, line thickness, and line type. It is considered that the stronger the connection, the greater the relevance between the material contents. In the example of FIG. 7, the high relevance between material contents CT1 and CT2, the relatively low relevance between material contents CT4 and CT8, and the lack of relevance between material content CT9 and other material contents are expressed by the number and type of lines indicating the connection, or the presence or absence of lines. Of course, the relevance evaluation value may also be expressed by, for example, the distance, direction of the arrangement position of each material content, or by adding numerical values to the lines indicating the connection.

[0103] For example, the evaluation unit 15 generates relevance presentation information for displaying such an image and provides it to the story generation unit 16. The story generation unit 16 can provide such relevance presentation information to the user side and have it presented to the user by the UI unit 22.

[0104] The user can, through the UI unit 22, check the display content as shown in FIG. 7 and can also modify the evaluation by operation. For example, it is possible to perform operations such as modifying the individual evaluation value of material content that is an image of an important scene or a favorite image of the user itself to a high value, operating on material content that is related but has a low evaluation of relevance so that it has a higher relevance evaluation value, or cutting a line that is evaluated as relevant but is actually irrelevant. Such operation information is fed back to the server device 10 as the feedback information dFB shown in FIG. 5, enabling the evaluation information by the evaluation unit 15 to be corrected. Also, such feedback information dFB is input to and learned by the learning unit 17, and is reflected, for example, in a function for selecting material content by the story generation unit 16.

[0105] In the example of FIG. 7, individual evaluation values and relevance evaluation values are expressed for each material content, but it is also possible to express only the relevance based on the relevance evaluation value, or to present each individual evaluation value. That is, the relevance presentation information may be anything that can present some information such as evaluation and relevance regarding the material content to the user. Also, it is desirable to form a user interface that enables the user to perform an operation to provide the feedback information dFB for all or part of the presented content.

[0106] The above FIG. 7 shows an example of the relevance between material contents, but various display modes of relevance are conceivable. FIG. 8 is an example of presenting key elements and showing the material content related to them. In the case of FIG. 8, "joy", "anger", "sorrow", and "happiness" are used as key elements, and an example is shown where the material contents CT1, CT2 corresponding to "joy", the material contents CT7, CT8 corresponding to "anger", the material content CT3 corresponding to "sorrow", and the material contents CT4, CT5, CT6 corresponding to "happiness" are presented.

[0107] FIG. 9 is also an example of presenting key elements. In this case, a certain event (e.g., a piano recital) is used as a key element, and related material contents CT1, CT2, CT3, CT4, CT5 for this event are presented. Also, in this example, although not directly corresponding to the event, material contents CT6, CT7, CT8 related to the material contents CT4, CT5 related to the event are also presented.

[0108] FIG. 10 is an example of using composite key elements. As key elements, "family", "friends", etc. in the human relationship with User A are presented, and "soccer" as a hobby of User A is also presented. This is an example of presenting material contents related to these human relationships and key elements, as well as the relevance between material contents.

[0109] As described above, there are various ways of presenting examples using relevance presentation information, but it is desirable that the user can easily recognize the relevance of each material content. Of course, in examples such as FIG. 8, FIG. 9, and FIG. 10, individual evaluation values may be displayed as described in FIG. 7, and the relevance may be expressed in different ways.

[0110] <4. Automatic production process including content complementation> A specific example of the content automatic production process performed by the function of the story generation unit 16 in the server device 10 will be described.

[0111] FIG. 11 shows the automatic production process of moving content by the information processing device as the server device 10. In step S101, the server device 10 makes a production start determination in response to a trigger. This trigger is the start determination process in response to a request received from the user as described in the workflow WF3 in FIG. 6, or the start determination process in response to an event or the like as described in the workflow WF5.

[0112] In step S102, the server device 10 performs a story setting process for the moving content to be produced this time. A detailed example of the story setting process is shown in FIG. 12.

[0113] In step S120 of FIG. 12, the server device 10 performs, for example, the story assembly process as illustrated in FIG. 4, and also performs the selection process of material content to be applied to each scene constituting the story. The assembly of a story is a process of setting the beginning, middle, climax, and end for an event with a certain theme and assembling settings such as chapters CH1 to CH5 in FIG. 4. Also, one or more pieces of material content to be used in each chapter are selected. For example, among the material contents stored in the DB corresponding to the users for whom moving content is to be produced, candidate contents that can be applied to the scenes of each chapter (assuming that one chapter is composed of one or more scenes) are extracted. In this case, for example, when the hobbies and preferences of the user, their family, etc. can be determined from the user information, it is effective to extract the material content that is an image matching the hobbies of the user or the like. Then, the material content to be used for each scene is selected from among the candidate contents. Of course, it is desirable to reflect the preferences of the user or the like in this selection. A more specific processing example of step S120 will be described later with reference to FIG. 21.

[0114] After performing the story setting and the selection of corresponding material content in the process of step S120, the server device 10 determines in step S121 whether there is a shortage of material content. If it is determined in the process of step S120 that the story can be set and all the necessary material content can also be prepared, the process proceeds to step S124, where it is determined that content supplementation is not required (for example, the supplementation flag is turned off), and the process of FIG. 12 ends. That is, the process proceeds from step S102 to step S103 in FIG. 11.

[0115] In this case, since the supplementation flag is off, the server device 10 determines in step S103 that material content supplementation is not required and proceeds to step S105 to perform the editing process. In this editing process, the server device 10 uses the chapter configuration set in the story setting process and the selected material content to generate moving content that constitutes a series of stories. For example, from each selected video as material content, necessary scenes are cut out, image quality is adjusted, image effects are added, and then the material content is joined together according to the progress of the chapter.

[0116] When the moving content is completed through such an editing process, the server device 10 performs, as step S106, the process of providing the moving content to the user side. For example, it enables the user to view it by downloading a movie as moving content or through streaming. The above is the processing procedure in the case exemplified by the workflows WF3 and WF5 in FIG. 6.

[0117] On the other hand, there may be a case where it is determined in step S121 of FIG. 12 that the material content is insufficient. For example, there is no material content applicable to a certain scene. Or, although there is material content that conforms to a certain scene in terms of content, it may be determined that there is no appropriate material content because the evaluation value (individual evaluation value or relevance evaluation value) is low and it is not suitable for use, or the image quality is low.

[0118] However, even if some of the material content is insufficient, for example, if deleting that scene does not cause a major change in the story structure or if it is not a very important scene, there may be cases where the lack of material content can be tolerated. Also, there may be cases where information is input by the user intentionally determining that the material content for that scene is unnecessary. In such cases, the deficiency is also tolerated. In other words, in such cases, tolerating the deficiency, that is, intentionally omitting a certain scene, is considered to be more in line with the user's intention. When the server device 10 determines that the shortage of material content is within an acceptable range, it proceeds from step S122 to step S124 and then proceeds to step S105 in FIG. 11 as described above.

[0119] However, when there is a shortage of important material content, or when adopting a logic that does not allow a shortage of the material content required for the scene configuration, the server device 10 proceeds to step S123 in FIG. 12 and determines whether the story can be changed. For example, the server device 10 evaluates the importance of the story in the scene where the material content is insufficient, the possibility of partial story changes within the chapter, etc., and determines whether the story can be changed. The user's preference based on learning may also be added as a determination factor. For example, for the current user, if an evaluation such as "such a scene is not important" is possible, the story can be changed.

[0120] If the story can be changed, it returns from step S123 to step S120 and starts over with the story assembly and material content selection.

[0121] When it is determined that the material content is insufficient and this shortage cannot be tolerated in the current story and changing the story is not appropriate, the server device 10 proceeds from step S123 to step S125, determines that material content replenishment processing is required (for example, turns on the replenishment flag), and ends the processing in FIG. 12. That is, it ends step S102 in FIG. 11 and proceeds to step S103. In this case, with the replenishment flag being on, the server device 10 determines in step S103 that material content replenishment is required and proceeds to step S104 to execute the material content replenishment processing. That is, such a case is the situation described in the workflows WF4 and WF6 in FIG. 6.

[0122] An example of the material content replenishment processing performed in step S104 in FIG. 11 will be described. FIG. 13 is an example of a process of requesting an external device to record material content that constitutes a story as material content completion processing. In step S140, the server device 10 performs a process of requesting an external device to record material content, that is, to record an image or the like by shooting. The external device is each of the various devices shown in FIG. 1 and corresponds to the material content providing unit 20 in FIG. 2.

[0123] For example, the server device 10 transmits, as a message to the user, a request content such as "Please shoot a video of the scene of ○○" to the mobile terminal 5 or the like which is an external device. It is also conceivable to transmit a request message such as "Please shoot a video of the scene of ○○ of Mr. A" to the mobile terminal 5 or the like of the user's family or friends. Alternatively, the server device 10 may transmit control information for instructing imaging or providing a target scene to the application of the mobile terminal 5, the agent device 8, the robot, the monitoring camera 9, or the like.

[0124] That is, the material content request process in step S140 is a process of requesting or instructing an external device to record the material content of a necessary scene and transfer it to the server device 10 (so that it can be used as material content for creating moving content).

[0125] After performing the request process in step S140, the server device 10 waits for the input of the material content in step S141. While waiting for the input of the material content, the server device 10 checks for timeout or input of the material content in step S142, and waits for the timing of re-request in step S143. For example, a timeout is set when one week has passed since the request, and a re-request is made at a predetermined timing before the timeout occurs.

[0126] Before the timeout deadline, if it is time to re-request, the server device 10 returns to step S140 and performs the material content request process again. In this case, the same request process as the previous time may be used, or it is also conceivable to change the external device to which the request is sent or to change the content of the request.

[0127] When the material content is input in response to the request for the material content, or when the material content is not input even when the timeout deadline is reached, the server device 10 finishes the material content completion process from step S142.

[0128] For example, if the material content completion process as shown in FIG. 13 is performed as step S104 in FIG. 11, the process returns to the story setting process of step S102. In this case, the process described in FIG. 12 is re-executed. However, if the necessary material content can be obtained, the server device 10 may perform a process of filling the scene where the material content was insufficient with the newly obtained material content in step S120. In addition, when the material content completion process ends due to timeout, or when the input material content cannot be applied in terms of quality, or when it is determined that the content is not appropriate as the scene content, etc., it may be considered to start over from the assembly of the story.

[0129] For example, as a result of performing the process of step S120 in FIG. 12 through the material content completion process as shown in FIG. 13 and proceeding from step S121 or step S122 to step S124, the process proceeds to the editing process of step S105 as the process in FIG. 11, and the moving content is generated.

[0130] FIG. 14 shows another example of the material content completion process. This is an example of expanding the target for extracting the material content before requesting the material content from an external device.

[0131] In step S150, the server device 10 performs a process of searching for material content applicable to a scene lacking material content by referring to a DB associated with others suitable for the user of the current production target or a public DB based on the user information. For example, basically in step S120 of FIG. 12, it is assumed that material content is extracted from the DB associated with the user of the production target. And when the material content is insufficient, the target of searching for the material content is expanded to the DB about others or the public DB.

[0132] Others refer to the family, friends, colleagues, etc. of the target user based on the user information. The DB associated with others is a DB that manages images taken by people corresponding to these "others" in the service. Also, it is conceivable to search for material content taking into account the user's hobbies and preferences, for example, from the perspective of what kind of images the user and their family like. For example, when the hobbies and preferences of the user and their family can be determined from the user information, search for material content that is determined or estimated to be an image that matches the user's hobby, or a scene or image content that the user likes.

[0133] In step S151, the server device 10 determines whether it has discovered the material content required this time from the DB associated with others or the public DB. If it has been discovered, the material content complementation process ends and the process proceeds to step S102 of FIG. 11.

[0134] In step S151 of FIG. 14, if it is determined that the required material content has not been discovered, the server device 10 proceeds to step S152 and performs a process of requesting material content from an external device. Then, the processes of waiting for input in step S153, input of material content or timeout in step S154, and waiting for re-request in step S155 are performed. The processes from step S152 to step S155 are basically the same as the processes from step S140 to step S143 of FIG. 13.

[0135] However, in the material content request process in step S152, it is also conceivable to request material content in consideration of the hobbies and preferences of the user and their family members, for example, from the perspective of what kind of images the user likes. For example, if hobbies and preferences of the user and their family members can be determined from the user information, requests for shooting scenes and image content that match the user's hobbies, or that are determined or presumed to be liked by the user, are made. For example, for a user who has jogging as a hobby and is determined or presumed to like running, recommendation messages such as "Please be sure to take pictures of the relay" or "I recommend taking pictures near the goal" are sent at a children's sports meet. This makes it easier to obtain material content that the user likes.

[0136] As described above, before performing the material content request process for an external device, by expanding the range of the DB for searching material content, the possibility of quickly fulfilling the material content can be increased. By also using images taken by others, it can be expected that the range of content will expand, for example, an image with an angle that the user himself / herself did not intend will be inserted into the moving content.

[0137] FIG. 15 is also another example of the material content complement processing. In step S160, the server device 10 performs a process of searching for material content applicable to a scene where the material content is insufficient by referring to a DB associated with others or a public DB that matches the user of the current production target based on the user information. This is the same process as step S150 in FIG. 14.

[0138] In step S161, the server device 10 determines whether the necessary material content for this time has been discovered from the DB associated with others or the public DB. If it has been discovered, the material content complement processing is terminated and the process proceeds to step S102 in FIG. 11.

[0139] When it is determined that the necessary material content cannot be found, the server device 10 proceeds from step S161 to step S162 and performs the acquisition process of fake images and cut-out images. The fake image referred to here means that it is not an image actually taken of the subject, and refers to a composite image, a CG image, an animation image, etc. The server device 10 generates an image necessary for the scene by image synthesis processing, CG processing, etc. Alternatively, the server device 10 instructs an external device to execute such image generation and acquires the generated image. Alternatively, the server device 10 cuts out a part of the image of other material content, etc., and uses it as the material content adapted to the scene. For example, it is also assumed that a scene in which the corresponding user appears is cut out from the video of a surveillance camera or the imaging image of another person.

[0140] By performing these processes, the server device 10 generates the lacking material content and uses it for the corresponding scene. After completing the material content supplementation process through the process of step S162, the server device 10 proceeds to step S102 in FIG. 11.

[0141] Note that the process of step S162 in FIG. 15 may be executed when a time-out occurs, for example, when the material content request process for the external device is performed in FIG. 13 or FIG. 14.

[0142] <5. Pre-production Recommendation / Complementation Process> Although the processes from FIG. 11 to FIG. 15 have been described as the processes after starting the production of the moving content, it is also possible for the server side to seize the opportunity to produce the moving content and recommend image imaging, etc. to the user. An example of the process in such a case will be described.

[0143] FIG. 16 is an example in which the server device 10 determines the recommendation timing for the user based on the user information and performs the material content supplementation process.

[0144] In step S201, the server device 10 determines the recommended timing for acquiring materials for moving content according to events that will occur in the future. Similar to the production triggers described in the workflows WF5 and WF6 of FIG. 6, for example, based on event information, schedule information, etc., events that provide opportunities for producing moving content are checked. Then, it is determined whether it is a recommended opportunity for shooting images or the like used for producing such moving content. Also, even if it is not an event or the like, the recommended opportunity may be determined according to the user's life stage. For example, information such as age progression, changes in family composition, or marriage and childbirth is checked, and milestones in the life stage are regarded as opportunities for producing moving content for commemoration, and the timing close to that is determined as the recommended opportunity.

[0145] For example, the server device 10 determines the recommended timing as the time point before an event that can be a material, such as one week before a child's sports meet or three days before the wedding of the target user, for a certain target user.

[0146] After determining the recommended timing, the server device 10 proceeds from step S202 to step S203 and performs material content complement processing for future production of moving content. This material content complement processing is basically a process of requesting the user to take pictures of such scenes in events that the user will participate in from now on.

[0147] An example of the material content complement processing in step S203 is shown in FIG. 17. In step S220, the server device 10 refers to the user information of the recommendation target and searches for matching content (material content or moving content) by referring to the DB of others or the public. For example, when it is the case that next week is the school sports meet for the target user, the server extracts the content of the sports meet of others. In this case, the "others" do not necessarily have to be the user's relatives, and it is advisable to widely refer to the DB of completely unrelated people.

[0148] In step S221, the server device 10 analyzes high-evaluation scenes, image content, etc. in the extracted content of others, and identifies moving elements. For the evaluation, for example, feedback information dFB for moving content, the evaluation value of material content, etc. can also be used, or in general criteria, scenes, shooting angles, etc. that are often used in the same event (for example, a sports meet) can be regarded as highly evaluated ones. For example, when it is the case that next week is the school sports meet for the target user, the server refers to the content of the sports meet of others, identifies high-evaluation scenes, image content, etc., and uses them as moving elements.

[0149] In step S222, the server device 10 generates shooting recommendation data based on the analysis in step S221. For example, according to high-evaluation scenes, angles, etc., shooting recommendation data is generated using text and image data as information including shooting methods, advice, etc.

[0150] Then in step S223, the server device 10 performs material content request processing. In this case, together with the recommended content, it becomes the process of transmitting the shooting recommendation data to the user's mobile terminal 5, etc. For example, together with a recommendation message such as "Next week is your child's sports meet. How about making some content?", shooting recommendation data generated with reference to the content of others, etc. as the shooting method at the sports meet is transmitted.

[0151] By performing such a recommendation prior to the event, it is expected that it will be easier for the user to provide more desirable material content, and the content of the moving content produced later will also be highly satisfactory. Particularly, since elements such as being able to refer to how others take pictures are included, it can provide useful information even for users who are not used to photography or the like in general.

[0152] FIG. 18 also shows an example of the process of step S203 in FIG. 16. In step S230, the server device 10 performs a process of assembling a temporary story based on event information for making a recommendation. Then, in step S231, the server device 10 sets what kind of material content is required as a scene corresponding to the set temporary story, and generates shooting recommendation data indicating the content of such a scene.

[0153] Then, in step S232, the server device 10 performs a material content request process. In this case, together with the recommendation content, as shooting recommendation data, it is a process of transmitting information presenting the content of the scene to the user's mobile terminal 5 or the like. For example, together with a recommendation message such as "Next week is your live stage. Won't you create content?", present the content of desirable scenes such as "venue scenery", "stage shot", "backstage scenery", "launch situation", etc. according to the set story.

[0154] By making such a recommendation prior to the event, it becomes easier for the user to provide material content according to the story, and as a result, it is expected that the content of the moving content to be produced later will also be highly satisfactory.

[0155] <6. Evaluation and Feedback of Material Content> Subsequently, the evaluation and feedback process of the material content in the server device 10 will be described. As described above with reference to FIG. 5, the evaluation unit 15 sets a relevance evaluation value and an individual evaluation value for the material content. Also, corresponding processing, that is, evaluation value update processing, is performed according to the feedback information dFB from the user.

[0156] The reason for setting and updating the evaluation value in this way is that when the server device 10 stores and manages a huge amount of content (memory videos and photos) as a large number of material contents for a large number of users, it can be appropriately used for content production. For example, since each material content is fragmentary and has no story, it is highly likely to be buried if only stored. Therefore, through AI processing, by mapping the analysis results with different viewpoints into a mining map, the degree of explicit expression and combination of content with high story value can be expressed as an evaluation value from the importance and relevance. This makes it easier to select valuable material content when generating moving content and improves the quality of the automatically produced moving content.

[0157] In addition to setting the evaluation value, it is visualized as relevance presentation information to enable user feedback, so as to make corrections to the relevance evaluation value and individual evaluation value, and update the evaluation function for producing moving content, etc., to be more suitable for the individual user.

[0158] First, FIG. 19 shows the material content evaluation process performed by the server device 10 according to the function of the evaluation unit 15.

[0159] In step S300, the server device 10 checks the input of feedback information dFB regarding the material content. If there is feedback information dFB, feedback corresponding processing is performed in step S320. This will be described later.

[0160] If no feedback information dFB is input, the server device 10 proceeds to step S301 to identify the material content to be processed. This is to identify, for example, material content that has been stored but not yet evaluated, such as newly uploaded material content, as the target of this evaluation process. However, for material content for which an evaluation value has already been set, it may also be targeted for re-evaluation in response to the passage of time, etc.

[0161] In step S302, the server device 10 calculates an individual evaluation value for the material content targeted for the evaluation process. As described above, the individual evaluation value is a value for evaluating the value of the image itself, the scene, the importance of the content, etc. of the material content itself. Therefore, the server device 10 performs image analysis, analysis of user information, etc., and determines the applicability of emotions such as joy, anger, sorrow, and pleasure, whether it was targeted for tracking (focus target) during imaging, whether the image content is a sudden event, whether it is an image with positive elements, whether it is an image with negative elements, the number of playback times as the user's viewing history, etc. as indicators, and calculates the individual evaluation value with a predetermined calculation. Also, when it is possible to determine the hobbies and preferences of the user and their family, etc. from the user information, it is desirable to reflect the hobbies and preferences of the user and their family, etc. in the evaluation value. For example, a high individual evaluation value is set for an image that matches the hobby, or an image or scene content that is determined or estimated to be preferred by the user.

[0162] In step S303, the server device 10 sets key elements for the material content. For example, key elements such as specific emotions, identification of related persons, event titles, etc. are set. These can be set by referring to the analysis results of the image content, user information, metadata added to the image, etc.

[0163] In step S304, the server device 10 extracts related material content using the key elements. For example, other material content taken at the same event as the material content targeted for the evaluation process, or other material content taken of the same specific related person (e.g., child) as the material content targeted for the evaluation process, etc. are extracted.

[0164] In step S305, the server device 10 calculates a relevance evaluation value for the material content targeted for the evaluation process. For example, the degree of relevance between the material content with common key elements is quantified to calculate the relevance evaluation value. The degree of relevance can be determined or estimated from the image content such as, for example, the causal relationship of events, the time-series relationship, and the existence of people or articles related to the events. Also, when the hobbies and preferences of the user and their family members can be determined from the user information, it is desirable to reflect the hobbies and preferences of the user and their family members in the evaluation value. For material contents that have a causal relationship with each other, a high relevance evaluation value is set between those that are determined or estimated to be images related to the user's hobbies or material contents that include scenes or image contents preferred by the user.

[0165] In step S306, the server device 10 generates or updates relevance presentation information. This will involve, for example, a process of newly generating relevance presentation information as described in FIGS. 7 to 10, or a process of incorporating the material content to be evaluated this time into the existing relevance presentation information.

[0166] In step S307, the server device 10 saves the calculated individual evaluation value, relevance evaluation value, and key elements for the material content to be processed. Thereby, the evaluation process is completed.

[0167] When feedback information dFB regarding the material content is input, the server device 10 performs the process shown in FIG. 20, for example, as the feedback response process in step S320.

[0168] In step S321, the server device 10 identifies the material content to be the feedback target. In step S322, the server device 10 determines whether it is necessary to correct the individual evaluation value for the material content identified as the feedback target based on the feedback information dFB. If necessary, the server device 10 proceeds to step S323 and performs an update process for the individual evaluation value.

[0169] In step S324, the server device 10 determines whether it is necessary to correct the individual evaluation value for the material content identified as the feedback target based on the feedback information dFB. If necessary, the server device 10 proceeds to step S325 and performs an update process on the key elements set for the material content.

[0170] In step S326, the server device 10 determines whether it is necessary to correct the relevance evaluation value for the material content specified as the feedback target based on the feedback information dFB. If necessary, the server device 10 proceeds to step S327 and re - sets the related material content for the material content. That is, it is other material content for which the relevance is presented in the relevance presentation information. In step S328, the server device 10 calculates and updates the relevance evaluation value between the newly set other related material content. In step S329, the server device 10 updates the relevance presentation information in response to the update of the other related material content and the relevance evaluation value between them.

[0171] By performing the process as shown in this Figure 20, the evaluation value set by the evaluation unit 15 of the server device 10 will be sequentially updated according to the feedback information dFB from the user.

[0172] Incidentally, as described above, the feedback information dFB from the user is reflected not only in the evaluation value of the material content but also in the production of the emotional content. With reference to FIGS. 21 and 22, the process of step S120 in the story setting process described above with reference to FIG. 12 will be described in detail, and an example of reflecting the feedback information dFB will be explained.

[0173] Although it was described that in step S120 of FIG. 12, the story is assembled and the material content corresponding to the story is selected, for example, the server device 10 specifically performs the process as shown in FIG. 21.

[0174] In step S401, the server device 10 assembles a story. The story is assumed to be assembled as in the example of FIG. 4. For example, story assembly is performed by arranging a configuration corresponding to the beginning, middle, climax, and end. Also, for example, when an interim story is assembled in step S230 of FIG. 18 for a certain event, story assembly may be performed based on that. This is because it is assumed that there is a possibility that material content suitable for the scene of that event is provided by the user.

[0175] In step S402, the server device 10 collects candidate content. That is, for each chapter constituting the story, one or more candidate contents to be used are collected from the DB etc. of the user who is the production target. At this point, multiple material contents may be used as candidate contents for use in one scene. For example, candidate content is collected based on scene content, user information, metadata, etc., regardless of the evaluation value.

[0176] In step S403, the material content to be used in each scene constituting the story is selected. For example, for one scene, a method of selecting the material content with the highest individual evaluation value among a large number of candidate contents may be used, but it is desirable to more comprehensively reflect relevance and the like. For example, in addition to selecting the material content with the highest individual evaluation value for each scene, the relevance evaluation value between the selected material contents is also reflected, and the multiple material contents as the selection result are optimized. Also, the combination, order, duration, etc. of the material content are elements of the selection. The selection process of the material content to be used in each scene constituting the story is performed using an evaluation function including elements such as such individual evaluation values, relevance evaluation values, duration, and order.

[0177] Basically, material content with high individual evaluation values and relevance evaluation values is likely to be selected, but updates according to the feedback information dFB based on user operations as described in FIGS. 19 and 20 are added to these evaluation values, so that material content suitable for the user's preference is likely to be selected. Moreover, when initially setting the individual evaluation values and relevance evaluation values in Fig. 19, by performing evaluation value setting that reflects the preferences of the user and their family, etc. (such as what kind of images and scenes they like), which can be determined or estimated from the user information, it becomes easier to reflect the user's preferences in the material content applied to the scene.

[0178] In step S404, based on the selection result of the material content in step S403, the server device 10 generates editing data. The editing data shall include information on the selected material content, information on the order of splicing the material content, information on the processing (such as image quality adjustment, cutting, effects, etc.) for each material content, etc. That is, it is the design information for creating moving content.

[0179] For example, such processing is performed as step S120 in Fig. 12, and later, the editing process based on the editing information is executed in step S105 of Fig. 11.

[0180] Here, when feedback information dFB based on a user operation is input, the processing in Fig. 22 can be performed by the functions of the learning unit 17 and the story generation unit 16.

[0181] When the feedback information dFB is input to the server device 10, the server device advances the processing in Fig. 22 from step S420 to step S421. If the feedback information dFB requests a change to the story of the moving content currently being produced, it proceeds from step S421 to step S401 in Fig. 21, and the story assembly will be redone. For example, based on the feedback information dFB, it is assumed that some scenes will be cut or replaced. Note that the feedback information dFB for the moving content being produced is assumed to be transmitted in response to a user operation, for example, when the server device 10 allows the user to have a preview before the final delivery of the moving content. When the feedback information dFB requires a change in the current material content, the server device 10 proceeds from step S422 in FIG. 22 to step S403 in FIG. 21. This causes the selection of the material content in step S403 to be redone.

[0182] The feedback information dFB is not necessarily reflected in the moving content being produced. For example, there may be cases of feedback information dFB regarding the presentation by the relevance presentation information. However, since the operations by the user also affect the story composition, it is desirable to make it learn for future story generation. For example, if it is content that is to be reflected in future content production, the server device 10 proceeds from step S423 to step S424 and performs learning / updating processing of the evaluation function. For example, some coefficients or variable elements of the evaluation function used when producing moving content for the user are updated. As a result, for example, the direction of material content selection in step S403 in FIG. 21 and the elements to be emphasized are corrected. That is, the selection process of the material content is adapted according to the preferences of that user.

[0183] Note that above, mainly as functions of the evaluation unit 15 and the learning unit 17, an example of adapting the moving content to be generated by performing generation, learning / updating processing of the evaluation function has been described. However, instead of the above processing, it is also conceivable to use a learning model learned using machine learning or deep learning so that the moving content to be generated is adapted to the user. For example, the story generation unit 16 includes a function that substitutes for the evaluation unit 15, and performs moving content generation while performing evaluation, classification, and extraction of the material content. Specifically, the story generation unit 16 performs deep learning based on various types of information regarding the user, and generates a learning model that suits that user. Then, for each material content, a learning model is applied for classification, extraction, and further determination of whether it is a material content applicable to a story. Based on the determined relevant material content, moving content is generated.

[0184] In this way, the generation function of the learning model and the content generation function using the learning model are provided as an evaluation unit and a story generation unit 16. Also, as a learning unit 17, re-learning is performed based on the feedback information dFB of the user, and the learning model is updated, which is suitable for generating more desirable moving content.

[0185] <7. Management of Negative / Positive Content> There are materials that can be classified as negative content or positive content as material content. Negative content is generally not considered content that users would particularly want to watch. However, considering its use in story composition as personal content as described in FIG. 4, negative content can be useful material content for enhancing the emotion during viewing. On the other hand, considering a form where, for example, the server device 10 functions as a cloud storage and provides a storage area for the user's personal memory in the storage 10a, negative content may also be easily viewable by the user like other material content and may be considered undesirable.

[0186] Therefore, in the server device 10, negative content is stored and managed separately from other material content by the function of the memory management unit 18. For example, a remote storage 12 with poor accessibility compared to the storage 10a is prepared where access from the user is not ensured or even from the server device 10, and negative content is stored there.

[0187] Figure 23 shows the material content management process performed by the server device 10 according to the function of the memory management unit 18. In step S501, the server device 10 identifies the material content to be processed.

[0188] In step S502, the server device 10 determines whether the material content to be processed is negative content. Whether it is negative content may be determined by some process. For example, if there is user vital information at the time of shooting as metadata attached to an image, it may be determined whether negative emotions are generated based on it, or it can also be estimated from the user's expression, action, gesture, etc. by image analysis. Regarding the fact that the user does not view it as a negative element, it may be added to the determination element. Also, regarding features different from other daily images as negative elements, they may be added to the determination element. Also, in the case where there is an interval between daily images, it is conceivable to estimate that some negative event has occurred, and add it to the estimation element on the assumption that negative elements are likely to exist in the images during such interval periods.

[0189] Note that positive content can also be determined by a method similar to the determination of such negative content.

[0190] If it is determined in step S502 that it is not negative content, the server device 10 proceeds to step S503 and performs normal storage for the material content. For example, it is stored in a DB associated with the corresponding user in the storage 10a.

[0191] If it is determined in step S502 that it is negative content, the server device 10 proceeds to step S504 and performs storage processing corresponding to negative content for the material content. For example, it is stored in a DB associated with the corresponding user in the remote storage 12.

[0192] In the case of negative content like this, it is saved separately from other material content. Although the saving to the remote storage 12 is cited as an example of the saving corresponding to the negative content in step S504, the following storages can be considered as this remote storage 12.

[0193] · A storage device arranged remotely separately from the storage 10a · A storage device with lower accessibility from the server device 10 than the storage 10a · A storage device managed so that access from the user is restricted · A storage device accessible via a communication path with a low transfer rate · A storage device connected to a stand-alone device · While the storage 10a is connected from the server device 10 via the first network, the remote storage 12 is a storage device connected from the server device via a second network with a slower communication speed than the first network.

[0194] Although the above is an example, it can be considered that the remote storage 12 is assumed to be not normally accessed.

[0195] Also, the process of step S504 is not necessarily limited to recording on the remote storage 12. For example, negative content may be saved in the storage 10a in the same way as other material content. However, hierarchical management is performed so that its access becomes difficult, encryption is performed, or access permission settings are made only for specific authorized persons, etc., to distinguish the accessibility.

[0196] Although the above process of FIG. 23 is an example of saving and managing negative content separately from other material content, negative content, positive content, and other material content may be managed separately. In that case, the negative content reduces accessibility by, for example, separating storage, but the positive content and other material content may have the same accessibility. For example, the positive content and other material content are similarly stored in the corresponding user's DB in storage 10a, but in the case of positive content, a flag indicating that it is positive content may be set. Also, all material content may be divided into negative content and positive content. For example, those that do not correspond to negative content may be managed as positive content.

[0197] As described above, negative content is managed separately, but since negative content is useful when it is desired to be used as material for creating moving content, it is desirable to change the management state according to the situation. Therefore, it is conceivable to perform the processing as shown in FIG. 24.

[0198] The sequential storage process in step S520 indicates a process in which, for example, when material content is uploaded, the server device 10 sequentially stores the material content by the process shown in FIG. 23.

[0199] Here, in step S521, the server device 10 checks whether the material content saved this time is positive content. If it is not positive content, that is, if it is negative content or material content that is neither negative content nor positive content, the process of FIG. 24 ends.

[0200] If it is positive content, the server device 10 advances the process to step S522 and searches for negative content related to the current positive content. For example, if the current positive content is such that the user was active in a baseball game, a process is performed to search for negative content of a scene where the user was injured during practice in the past.

[0201] If the corresponding negative content cannot be found, the process ends from step S523. However, if the corresponding negative content is found, the server device 10 performs a process of changing the management state of the negative content in step S524. For example, a process is performed to improve the accessibility for the server device 10, such as moving the storage device from the remote storage 12 to the storage 10a.

[0202] Note that the negative content managed as negative content correspondence storage in step S504 of FIG. 23 may not be entered as candidate content in the process of the story generation unit 16. That is, negative content is not used for creating moving content unconditionally. However, as the process of step S524, when corresponding positive content exists, it is conceivable to change the management state so that the negative content is given the qualification to be candidate content. That is, it can be used for creating moving content only when the importance arises in the story of highlighting the positive content.

[0203] In step S525, the server device 10 updates the evaluation value of the negative content in response to the occurrence of relevance with the positive content saved this time. That is, the relevance evaluation value and the individual evaluation value are recalculated and updated.

[0204] Through the above processing, negative content corresponding to positive content can be caused or enhanced to be applicable to a certain scene in the production of moving content. Thereby, it can be expected to enhance the story quality of moving content automatically produced using appropriate negative content.

[0205] FIG. 25 is also a processing example with the same purport. The sequential saving process in step S530 indicates that, for example, in response to the upload of material content, the server device 10 sequentially saves the material content by the process shown in FIG. 23, for example.

[0206] Here, in step S531, the server device 10 checks whether the material content saved this time is negative content. If it is negative content, in step S532, a process of registering a negative event is performed. For example, for a certain user, information such as when and what negative event occurred is registered as one of the user information.

[0207] If it is determined in step S531 that the content is not negative content, the server device 10 determines in step S533 whether the material content saved this time is positive content corresponding to a negative event registered in the past. If the material content saved this time is positive content related to a negative event registered in the process of step S532 in the past, the server device 10 proceeds to step S534 and performs a process of changing the management state of the negative content related to the registered negative event. For example, a process of improving the accessibility for the server device 10, such as moving the storage device from the remote storage 12 to the storage 10a, is performed. Also in this case, it is conceivable that the server device 10 changes the management state so that the negative content can be given the qualification to be candidate content.

[0208] In step S535, when the server device 10 determines that there is a relevance between the negative content and the positive content saved this time for the negative content, the server device 10 updates the evaluation value of the negative content. That is, the relevance evaluation value and the individual evaluation value are recalculated and updated.

[0209] Through the above processing, the negative content corresponding to the positive content can be made to have the possibility of being applied to a certain scene in the production of moving content, or the possibility can be enhanced.

[0210] <8. Summary and Variation Examples> According to the above embodiments, the following effects can be obtained. The server device 10 as the information processing device in the embodiment includes a story generation unit 16 that performs a process of requesting an external device to record material content that constitutes a story for the moving content to be automatically produced (see FIGS. 11 to 18). This process is useful for collecting materials for moving content that features the personal actions, experiences, etc. of customer users in the story. In particular, by making a specific request such as what scenes are desired to be photographed, it is suitable for collecting material content.

[0211] In the embodiment, an example is given in which the story generation unit 16 performs a story setting process for moving content, selects the material content to be used, determines the shortage of the material content, and performs a process of requesting the external device to record the material content in response to determining the shortage of the material content (see FIGS. 11, 12, 13, and 14). By collecting material content according to the story and requesting recording from the external device in case of shortage, the material content complementing process can be efficiently executed in the production of automatically produced content.

[0212] In the embodiment, an example is given in which the story generation unit 16 performs a recommendation timing determination process for a user regarded as a customer, and in response to the recommendation timing determination process, performs a process of requesting the external device to record material content (see FIG. 16). That is, the opportunity for automatic content production for a user regarded as a customer of the content automatic production service is determined based on the user's event schedule, life stage, etc. Then, a process of requesting the recording of material content is performed according to the determined timing. For the user regarded as a customer, by making recommendations such as image recording by shooting to the user or requesting recording to various devices according to events and life changes that will occur in the future, it becomes easier to obtain material content for opportunities that contribute to memories and emotions for the user. By using such material content, the possibility of providing automatically produced content with high user satisfaction can be increased. Note that the timing regarding the user regarded as a customer includes not only the user himself / herself but also the timing based on various events and life stages related to family and friends.

[0213] In the embodiment, an example is given in which the story generation unit 16 determines the user's preference based on the user information, and based on the determined preference information, performs a process of requesting the recording of material content. For example, the preference is determined based on the user information regarding the user himself / herself, his / her family, and friends regarded as customers, and based on this, a request for recording material content is made (see FIG. 14). By making a recording request based on the user information about the user himself / herself, family, relatives, friends, etc. regarded as customers, it becomes easier to collect material content that matches the preferences of the user on the viewer side, and the possibility of realizing automatically produced content according to the preferences can be increased.

[0214] In the embodiment, an example was given in which the story generation unit 16 determines the preferences of the user based on the user information, and performs a process of selecting material content using the determined preference information as a selection element. For example, the preference is determined or estimated based on the user information about the user himself / herself who is regarded as a customer, his / her family members, and friends. By setting an evaluation value reflecting such preferences, a process of selecting material content using the preference information as a selection element is performed (see step S403 in FIGS. 19 and 21). Based on the user information about the user himself / herself who is regarded as a customer, his / her family members, relatives, friends, etc., it is possible to select material content that matches the preferences of the user on the viewer side according to the story. This can increase the possibility of realizing automatically produced content that matches the user's preferences.

[0215] In the embodiment, an example was given in which the story generation unit 16 performs a process of requesting recording of material content based on the event information about the user (see FIGS. 16 and 18). For example, an appropriate scene or the like corresponding to the type of event information of the user regarded as a customer is determined, and a request for recording of matching material content is made. As events, there are various types such as sports games, presentation meetings, weddings, children's sports meets, stages for performances and plays, etc., and desirable scenes and the like can be considered according to the event. By requesting shooting of such scenes, etc., the possibility of realizing more highly completed automatically produced content can be increased.

[0216] In the embodiment, there are cases where the story generation unit 16 performs a process of selecting material content based on the event information about the user. When a story corresponding to the event is temporarily assembled in step S230 of FIG. 18 and story assembly is performed based on this in step S201 of FIG. 21, the material content selected in step S403 is selected based on the event information. That is, selection of material content that matches an appropriate scene or the like corresponding to the type of event information of the user regarded as a customer is performed. This can increase the possibility of realizing highly completed automatically produced content according to events.

[0217] In the embodiment, as an example of the recommendation timing determination process, a timing determination based on the user's life stage was given (see FIG. 16). For example, as the user's life stage, timing determinations such as birth, enrollment, graduation, employment, marriage, birth of a child, growth of a child, retirement, etc. are made, and selection of suitable material content and recording requests are made accordingly. The progress and changes of the user's life stage have a narrative nature and are suitable as opportunities for producing automatically produced content. Therefore, by performing timing determination according to the life stage, it is possible to request shooting, etc. at a timing suitable for collecting suitable material content for automatically produced content. This can make it easier to collect material content desirable for automatic production.

[0218] In the embodiment, an example was given in which the story generation unit 16 generates recommendation data indicating the content of the material content for which recording is requested, based on the event that was determined as the timing for requesting recording in the timing determination process and the content of other users regarding the same event (see FIG. 17). For example, the event that was determined as the timing for requesting recording in the timing determination process is, for example, an event in the user's life stage (enrollment, graduation, employment, marriage, etc.) or various events. When it is determined that recording is requested due to these events, data for recommending the content of the shooting scene, etc. is generated based on the content of other people in the case of the same event. The server device 10 generates shooting recommendation data indicating recommended content with reference to the content of other people (material content and automatically produced content) in the same event or life stage in the user's events, life stage, etc. Then it is transmitted to the user's terminal device and presented to the user. As a result, the user can know what kind of shooting should be done for this event, and it becomes easier to obtain material content suitable for automatically produced content, and it is easier to realize automatically produced content with high user satisfaction.

[0219] In the embodiment, the story generation unit 16 performs story setting for the content automatically produced for the user regarded as the customer, selects the material content to be used from the DBs corresponding to the user, and performs story setting processing for determining the shortage of the material content. Then, in response to the determination that there is a shortage of the material content, the material content is searched from other DBs (for example, DBs corresponding to other users, public DBs, etc.), and when it is further determined that there is a shortage of the material content, an example of performing processing for requesting the external device to record the material content is given (see FIG. 14). As a result, not only still images, moving images, etc. simply uploaded by the customer user, but also images taken by other people, such as close people like friends and family, or complete strangers, and further public images, etc., can be widely searched for material content. Thereby, content can be automatically produced using a more diverse image stock. Also, by minimizing the recording request, it is possible to increase the possibility of enabling quick content production without bothering the user. That is, the efficiency of the material content complementation process can be improved.

[0220] The server device 10 as the information processing device of the embodiment includes an evaluation unit 15 that performs processing for setting an evaluation value for the material content recorded in an accessible state, and a story generation unit 16 that selects the material content constituting the story set in the story setting process with reference to the evaluation value and performs automatic content production by an editing process using the selected material content (see FIGS. 19 to 22). By making the material content be selected with reference to the evaluation value (individual evaluation value or relevance evaluation value) during the production process of the automatically produced content, it is possible to increase the possibility of using material content more desirable for the user, and it becomes possible to provide automatically produced content with high user satisfaction.

[0221] In the embodiment, an example is given in which the story generation unit 16 performs a process of collecting material content candidates for use according to the set story, and selects, using the evaluation values among the material content collected as candidates, the material content to be used for automatic content production (see FIG. 21). Based on the evaluation values given to each material content, it is possible to easily and appropriately select the material content suitable for the current automatic production among the candidate contents. For example, candidate content corresponding to a set scene can be collected based on the scene content. However, it is difficult to select the most optimal material content for the current automatic production content or the material content that can move the current user only based on the image content. Therefore, selection is made considering the evaluation values reflecting the relationship between material contents and the importance of the material contents themselves. It is considered that this enables more desirable content production.

[0222] In the embodiment, an example is given in which the evaluation unit 15 sets a relevance evaluation value indicating the relevance of the material content to other material contents (see FIG. 19). For example, when there is a relevance such as a causal relationship between a material content and other material contents, an evaluation value indicating the relevance to other material contents is set. Each material content is not evaluated only as a single entity, but by evaluating the relevance to other contents, the matching between the contents used within one automatic production content is optimized, and it is considered that automatic content production that can give more impact becomes possible.

[0223] In the embodiment, an example is given in which the evaluation unit 15 sets an individual evaluation value for the material content as a single entity (see FIG. 19). Even if there is a vast amount of content as each material content, their importance levels differ. The importance levels vary depending on the importance as a general scene quality and the importance due to scenes where the user's emotions are expressed. And it is a heavy processing burden to determine such importance from each candidate content during automatic production. Therefore, an evaluation value of the content itself is set in advance so that it can be selected using that evaluation value. By doing so, it is considered that the selection of the material content used in the automatically produced content can be made easily and appropriately, and it becomes possible to automatically produce content that gives more impact.

[0224] In the embodiment, an example was given in which the evaluation unit 15 sets relevance presentation information that presents the relevance of a material content with other material contents (see FIGS. 19, 7 to 10). For example, as illustrated in FIGS. 7 to 10, by generating relevance presentation information indicating the relevance of each material content, the existence and the relationship of the material contents can be presented to the user. It becomes easier for the user to recognize the type and shortage of the material content.

[0225] In the embodiment, an example was given in which the relevance presentation information and the relevance evaluation value are updated according to a user operation (see FIG. 20). For example, by presenting the relevance presentation information in a form as illustrated in FIGS. 7 to 10, the user can confirm the appropriateness of the relevance between the material contents. And in practice, operations such as reducing the relevance between material contents with low relevance are possible, and the relevance information can be corrected by performing update processing according to the feedback information dFB. As a result, the relevance evaluation value and the relevance presentation information can be converged to more appropriate ones suitable for the user. Note that by also updating the individual evaluation value according to the feedback information dFB, it will converge to a state that reflects the user's hobbies and preferences.

[0226] The relevance presentation information in the embodiment shows the relevance between the material contents by directly connecting one material content and other material contents. For example, the relevance between contents is presented on the user interface by an image that connects material contents to each other (see FIGS. 7 to 10). As a result, the user can easily confirm whether the relevance between material contents is appropriate or not.

[0227] In the embodiment, an example was given in which the relevance presentation information indicates the relevance of material contents in a form in which material contents are associated with key elements (see FIGS. 8 and 9). For example, words or events indicating emotions are used as key elements, and the relevance of contents is presented on the user interface in such a form as material contents corresponding to a certain emotion, material contents related to a certain event, and the like. As shown in FIG. 8, by showing the relationship of material contents with the emotions of joy, anger, sorrow, and pleasure as key elements, or by providing an image showing the relationship of material contents with a certain event as a key element as shown in FIG. 9, the user can easily confirm the relevance of material contents according to his or her own emotions, events, etc., and whether they are appropriate or not. As other key elements, various considerations are possible, such as by life stage, specific keywords, types of actions, time, season, place, user age, and the like.

[0228] In the embodiment, an example was given in which the relevance presentation information indicates the relevance of material contents in a form in which material contents are associated with the user's human relationships (see FIG. 10). For example, as shown in FIG. 10, material contents related to family members and material contents related to friends are presented to user A. As a result, the user can easily confirm the relevance of material contents according to his or her own human relationships and whether they are appropriate or not. As elements of human relationships for presenting related material contents, various considerations are possible, such as, from the perspective of user A, parents, children, members of a certain circle, classmates, people at work, hobby friends, neighbors, and the like.

[0229] The server device 10 according to the embodiment includes a learning unit 17 that inputs a user operation on the relevance presentation information and performs a learning process. The learning unit 17 performs learning processing to reflect the feedback information dFB from the user on the relevance presentation information in the story setting. As a result, it becomes possible to create a story setting that reflects the user's intention, the importance and relevance of the material content, etc. in the future, or to correct the story of the automatically produced content that is currently being produced.

[0230] In the embodiment, an example in which the learning content by the learning unit 17 is reflected in the story correction is given. When proceeding from step S421 in FIG. 22 to step S401 in FIG. 21, the story of the automatically produced content that is currently being produced is corrected according to the learning result. As a result, the possibility of producing content suitable for the user can be increased.

[0231] In the embodiment, an example in which the learning content by the learning unit 17 is reflected in the selection of the material content is given. When proceeding from step S422 in FIG. 22 to step S403 in FIG. 21, the selection of the material content to be adopted according to the story of the moving content that is currently being produced is reselected according to the learning result. As a result, the possibility of producing automatically produced content using more suitable material content for the user can be increased.

[0232] In the embodiment, an example in which the learning content by the learning unit 17 is reflected in the future automatic production of content is given. For example, in steps S324 and S329 in FIG. 20, it is shown that the individual evaluation value and the relevance evaluation value are updated. Also, in step S424 in FIG. 22, it is shown that the evaluation function is updated so that the learning content is reflected in the future automatic production. As a result, the learning content will be reflected in the future automatic production of content. It is possible to evolve the automatic production system of moving content so that more desirable automatically produced content can be produced for the user.

[0233] As a server device 10 as an information processing apparatus according to an embodiment, a storage management unit 18 is provided that stores negative content determined to be negative scene content among material contents that constitute a story of automatically produced content in a state where it can be distinguished from other material contents (see FIGS. 23 to 25). During the production process of moving content, material content is selected according to the set scene. However, it is considered that more moving content can be generated by appropriately selecting negative content and arranging it in the story configuration. Since negative content is distinguished and stored and managed from other material content, negative content can be easily and appropriately selected during the production of such automatically produced content. In the embodiment, an example of distinguishing and managing negative content and positive content has been described. However, it may be configured to determine negative content and distinguish and manage negative content from other material content.

[0234] In the embodiment, an example has been given in which the storage management unit 18 performs a process of storing material content by distinguishing between negative content and positive content determined to be positive scene content (see FIGS. 23 to 25). In the scene set during the production process of moving content, it is desirable that positive content and negative content are appropriately selected. Since negative content and positive content are distinguished and stored and managed, negative content and positive content can be easily and appropriately selected during the production of such automatically produced content.

[0235] In the embodiment, an example has been given in which the storage management unit 18 stores the material content determined to be negative content in a remote storage 12 (second storage device) different from a storage 10a (first storage device) that stores and manages material content other than negative content. By storing negative content and other content (positive content) on separate storage devices, they can be appropriately distinguished, and it becomes possible to make negative content less visible to users, etc., under normal circumstances. For example, it is assumed that material content managed by a cloud server can be viewed not only as content for automatic production but also simply as videos, still images, etc. that a user saves. However, for a user, negative content is usually not something they want to look back at often, and in some cases, they may even want to forget it. There may also be cases where the image of the user was taken by someone other than the viewing user (such as a robot). In such cases, the user may not want to be seen by themselves or others. Therefore, it is stored in a separate storage and managed so that it is not normally viewed.

[0236] In the embodiment, it is assumed that storage 10a is a storage device managed in a state where user access to material content is permitted, and remote storage 12 is a storage device managed in a state where access to material content is permitted only to a specific administrator. For example, storage 10a and remote storage 12 shall have different access rights. By doing so, negative content can usually be made invisible to users.

[0237] In the embodiment, an example was described where remote storage 12 is a storage device with a slower access speed from the memory management unit 18 than storage 10a. For example, storage 10a and remote storage 12 shall have different access speeds due to differences such as inside and outside the network and communication methods. By doing so, negative content can be made less visible to general users. In order to reduce the communication speed, for example, when the storage 10a is a storage device within the device of the server device 10, or a LAN-connected storage device, or a storage device connected via a commonly used network, the remote storage 12 is assumed to be a storage device connected to a network that performs communication at a low transfer rate, a storage device located outside the city in a desert area, or a storage device located outside the earth such as the moon surface, a storage device connected to a stand-alone computer, etc. In any case, a storage device that is inconvenient for normal browsing can be assumed.

[0238] In the embodiment, an example is assumed in which the storage 10a is accessible by the storage management unit 18 via the first network, and the remote storage 12 is accessible by the storage management unit 18 via the second network. By differentiating the networks, the accessibility to negative content can be made different from that of other material content.

[0239] In the embodiment, an example is described in which the storage management unit 18 stores the material content determined to be negative content and the material content other than the negative content in the storage device in different hierarchical management states. For example, using the storage device of a server that manages material content, negative content and other material content are managed in different hierarchical folders. Or when using different storages, the hierarchical management states are made different. Also by hierarchical management, the accessibility to negative content can be made lower than that of others (positive content).

[0240] In the embodiment, an example is given in which the storage management unit 18 performs a process to facilitate access by the storage management unit 18 in response to the determination that the negative content is related to the positive content (see FIGS. 24 and 25). For example, negative content is usually stored and managed in a state where its accessibility is reduced. However, when the existence of related positive content is confirmed, it is managed in a state where its accessibility is improved. Negative content associated with existing positive content may be selected as one of the scenes constituting the moving content, making it easier for the server device 10 to access and facilitating its provision as candidate content during the production of automatically generated content. As a result, negative content that is usually managed in a state where it is difficult to view can be made more accessible for use in moving content.

[0241] In the embodiment, it has been described that the memory management unit 18 makes negative content into material content that can be used as candidate content in the content automatic production process when it is determined that the negative content is related to positive content. That is, usually, negative content is not considered a candidate for use in automatically generated content. However, when the existence of related positive content is confirmed, it can be made into candidate content. This can prevent negative content from being used in moving content without reason. Also, by making it possible to use negative content in content automatic production only when there is corresponding positive content, it will be used as content for a story that leads to subsequent positive events rather than just a negative scene. This enables the effective use of negative content.

[0242] In the embodiment, an example was described in which moving image content as material content was mainly used to automatically produce moving content such as a movie. Of course, still image data, audio data, text data, etc. may be used as material content, or these data may be used in combination. As an example of automatically produced content, a personal movie of a customer user was described as "moving content," but it is not necessarily limited to such personal moving content. Content of various types, such as content about a group or an organization, or content such as a reference video or an academic video, can be widely considered as "automatically produced content" as referred to in the present disclosure.

[0243] The program of the embodiment is a program that causes a device such as a CPU, a DSP, or the like, or a device including these, to execute the processing described in the embodiment.

[0244] The program of the embodiment is a program that causes an information processing apparatus to execute a process of requesting an external device to record material content that constitutes a story for content to be automatically produced. Also, the program of the embodiment is a program that causes an information processing apparatus to execute a process of setting an evaluation value for material content recorded in an accessible state, and a story generation process of automatically producing content by selecting material content that constitutes the story set in the story setting process with reference to the evaluation value and performing an editing process using the selected material content. Also, the program of the embodiment is a program that causes an information processing apparatus to execute a storage management process of storing negative content determined to be negative scene content among material content that constitutes the story of content to be automatically produced in a state where it can be distinguished from other material content.

[0245] Such a program can realize the above-described server device 10 in a computer device or other device capable of executing information processing.

[0246] A program for implementing such a server device 10 can be pre-recorded in an HDD as a recording medium built into a device such as a computer device, or in a ROM in a microcomputer having a CPU, etc. Alternatively, it can be temporarily or permanently stored (recorded) in a removable recording medium such as a flexible disk, CD-ROM (Compact Disc Read Only Memory), MO (Magneto Optical) disk, DVD (Digital Versatile Disc), Blu-ray Disc (registered trademark), magnetic disk, semiconductor memory, memory card, etc. Such a removable recording medium can be provided as so-called packaged software. Also, such a program can be installed from a removable recording medium into a personal computer or the like, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0247] Also, according to such a program, it is suitable for widespread provision of the server device 10 of the embodiment. For example, by downloading the program to a portable terminal device such as a smartphone or tablet, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), etc., the smartphone or the like can be made to function as the server device 10 of the present disclosure.

[0248] Note that the effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0249] Note that the present technology can also adopt the following configuration. (1) A storage management unit that stores, in a distinguishable state from other material contents, negative contents determined to be negative scene contents among the material contents that constitute the story of the content to be automatically produced. Information processing apparatus. (2) The memory management unit performs a process of separately storing, for material content, the negative content and the positive content determined as positive scene content. The information processing apparatus according to (1) above. (3) The memory management unit stores the material content determined as the negative content in a second storage device different from the first storage device that manages the material content other than the negative content. The information processing apparatus according to (1) or (2) above. (4) The first storage device is a storage device managed in a state where user access to the material content is permitted. The second storage device is a storage device managed in a state where access to the material content is permitted only to a specific administrator. The information processing apparatus according to (3) above. (5) The second storage device is a storage device whose access speed from the memory management unit is slower than that of the first storage device. The information processing apparatus according to (3) or (4) above. (6) The first storage device is accessible by the memory management unit via a first network, and the second storage device is accessible by the memory management unit via a second network. The information processing apparatus according to any one of (3) to (5) above. (7) The memory management unit stores the material content determined as the negative content and the material content other than the negative content in a storage device in different hierarchical management states. The information processing apparatus according to any one of (1) to (6) above. (8) When it is determined that the negative content is related to positive content determined as positive scene content, the memory management unit performs processing to facilitate access by the memory management unit. The information processing apparatus according to any one of (1) to (7) above. (9) When it is determined that the negative content is related to positive content determined as positive scene content, the memory management unit makes the negative content into material content that can be a candidate content for use in content automatic production processing. The information processing apparatus according to any one of (1) to (8) above. (10) Among the material contents that make up the story of the content to be automatically produced, the memory management process of storing the negative content determined as negative scene content in a state where it can be distinguished from other material contents. An information processing method executed by an information processing apparatus. (11) Among the material contents that make up the story of the content to be automatically produced, the memory management process of storing the negative content determined as negative scene content in a state where it can be distinguished from other material contents. A program to be executed by an information processing apparatus.

Explanation of Signs

[0250] 2 Cameras 3 Computer Devices 4 Tablet Terminals 5 Mobile Terminals 6 Camera - Equipped Aircraft 7 Camera - Equipped Vehicles 8 Agent Devices 9 Surveillance Cameras 10 Server Devices 10a Storage 11 Networks 12 Remote Storage 15 Evaluation Units 16 Story Generation Units 17 Learning Department 18 Memory Management Department 20 Material Content Provision Department 21 User Information Provision Department 22 UI Department

Claims

1. Among the material contents that make up the story of the content to be automatically produced, there is a storage management unit that stores the negative content determined to be negative scene content in a state where it can be distinguished from other material contents. The storage management unit stores the material content determined to be the negative content in a second storage device different from the first storage device that stores and manages the material content other than the negative content. An information processing apparatus.

2. The storage management unit performs a process of separately storing the negative content and the positive content determined to be positive scene content for the material content. The information processing apparatus according to claim 1.

3. The first storage device is a storage device managed in a state where user access to the material content is permitted. The second storage device is a storage device managed in a state where access to the material content is permitted only to a specific administrator. The information processing apparatus according to claim 1.

4. The second storage device is a storage device whose access speed from the storage management unit is slower than that of the first storage device. The information processing apparatus according to claim 1.

5. The first storage device is accessible by the storage management unit via a first network, and the second storage device is accessible by the storage management unit via a second network. The information processing apparatus according to claim 1.

6. Among the material contents that make up the story of the content to be automatically produced, there is a storage management unit that stores the negative content determined to be negative scene content in a state where it can be distinguished from other material contents. The storage management unit stores the material content determined to be the negative content and the material content other than the negative content in the storage device in different hierarchical management states. An information processing apparatus.

7. Among the material contents that make up the story of the content to be automatically produced, there is a storage management unit that stores the negative content determined to be negative scene content in a state where it can be distinguished from other material contents. When it is determined that the negative content is related to positive content determined as positive scene content, the memory management unit performs processing to facilitate access by the memory management unit Information processing apparatus. **Claim 8**: Among the material contents that make up the story of the content to be automatically produced, a memory management unit is provided that stores the negative content determined to have negative scene content in a state where it can be distinguished from other material contents. When it is determined that the negative content is related to positive content determined as positive scene content, the memory management unit makes the negative content into material content that can be a candidate content for use in content automatic production processing Information processing apparatus. **Claim 9** When storing the negative content determined to have negative scene content among the material contents that make up the story of the content to be automatically produced in a state where it can be distinguished from other material contents, A memory management process is performed so that the material content determined to be the negative content is stored in a second storage device different from the first storage device that manages the storage of material content other than the negative content An information processing method executed by an information processing apparatus. **Claim 10** When storing the negative content determined to have negative scene content among the material contents that make up the story of the content to be automatically produced in a state where it can be distinguished from other material contents, A memory management process is performed so that the material content determined to be the negative content is stored in a second storage device different from the first storage device that manages the storage of material content other than the negative content A program for causing an information processing apparatus to execute.

Images