Video creation assistance device and video creation assistance method

Abstract

A video creation assistance device according to the present invention comprises a reception unit that receives a plurality of pieces of video information that include, for a video that includes a time series of scenes, information about a viewing user, information about each scene, and information about enthusiasm for each scene, a model creation unit that performs machine learning using the information about the viewing user and the information about each scene as explanatory variables and the information about enthusiasm for each scene as an objective variable and creates a machine learning model that predicts information about enthusiasm, a prediction unit that inputs a plurality of pieces of information about a target user and information about a target scene for which enthusiasm is to be predicted into the machine learning model to predict information about enthusiasm for each target scene, and a recommendation unit that recommends a video that includes one or a plurality of the target scenes on the basis of prediction results from the prediction unit.

Description

Video creation support device and video creation support method

[0001] One aspect of the present disclosure relates to a video creation support device and a video creation support method.

[0002] Patent Document 1 discloses a technique for creating (assisting in the creation of) a composite video in consideration of user preference information.

[0003] Japanese Patent Application Laid-Open No. 2020-129189

[0004] In the field of video creation, there is a growing need to create popular videos (videos that users can get excited about) in a short amount of time.

[0005] The present disclosure has been made in consideration of the above-described circumstances, and provides a video creation support device and a video creation support method that enable users to create exciting videos in a short amount of time.

[0006] A video creation support device according to one aspect of the present disclosure includes a reception unit that receives multiple pieces of video information relating to a video including multiple scenes in chronological order, the video information including information about the viewing user, information about each scene, and information about the excitement in each scene; a model creation unit that performs machine learning on the multiple pieces of video information using information about the viewing user and information about each scene as explanatory variables and information about the excitement in each scene as a target variable, to create a machine learning model that predicts information about the excitement; a prediction unit that predicts information about the excitement in each target scene by inputting multiple pieces of information about the target user and information about the target scene for which the excitement is to be predicted into the machine learning model; and a recommendation unit that recommends videos that include one or more target scenes based on the prediction results by the prediction unit.

[0007] In a video creation support device according to one aspect of the present disclosure, a machine learning model is created using information about the viewing user and information about each scene as explanatory variables and information about the excitement in each scene as a target variable. Then, multiple pieces of information about the target user and information about the target scenes are input into the machine learning model, and information about the excitement in each target scene is predicted. Based on the prediction results, a video containing one or more target scenes is recommended. With this configuration, a machine learning model trained by associating information about the viewing user and the scenes with information about the excitement can be used to accurately predict information about the excitement in each target scene for the target user. By recommending videos containing one or more target scenes based on these prediction results, videos that are likely to excite the target user can be easily and appropriately recommended. As described above, the video creation support device according to one aspect of the present disclosure allows users to create videos that will excite them in a short period of time.

[0008] According to one aspect of the present disclosure, a video that can get users excited can be created in a short amount of time.

[0009] FIG. 1 is a diagram illustrating an overview of a video creation support device according to this embodiment. FIG. 2 is a diagram illustrating a functional configuration of the video creation support device according to this embodiment. FIG. 3 is a diagram illustrating an overview of creating a machine learning model. FIG. 4 is a diagram illustrating details of creating a machine learning model. FIG. 5 is a diagram illustrating details of excitement prediction and video recommendation. FIG. 6 is a flowchart showing processing executed by the video creation support device. FIG. 7 is a diagram illustrating an example of a hardware configuration of the video creation support device.

[0010] Hereinafter, the embodiments will be described in detail with reference to the drawings. In the description, the same elements or elements having the same functions are denoted by the same reference numerals, and redundant description will be omitted.

[0011] FIG. 1 is a diagram illustrating an overview of a video creation support device according to this embodiment. The video creation support device is a device that supports the creation of videos. Specifically, the video creation support device supports the creation of exciting videos by recommending videos including scenes that are predicted to be exciting when viewed by target users. The video creation support device creates a machine learning model that predicts information about excitement by learning a large amount of information about the excitement of viewing users for videos including multiple scenes in chronological order, and uses the machine learning model to predict the excitement of target users for each scene and recommend videos based on the prediction results.

[0012] In the graphs shown in Figures 1(a) and 1(b), the horizontal axis represents time and the vertical axis represents excitement level. Such excitement level is quantified, for example, by the number of times a scene is repeated (details will be described later). Figure 1(a) shows the excitement levels of women in their 30s and men in their 20s for video X, which includes multiple scenes in chronological order. The video creation support device creates a machine learning model that predicts information about excitement levels for a target user by associating and learning information about such viewers (e.g., women in their 30s), information about each scene (such as information about the actors and lines in each scene), and information about the excitement level in each scene (such as the excitement level according to the number of repetitions). Then, as shown in Figure 1(b), the video creation support device predicts information about the excitement level of the target user for each scene of video Y, which is different from video X, using the above-mentioned machine learning model and recommends videos taking the prediction results into consideration (details will be described later).

[0013] Fig. 2 is a diagram showing the functional configuration of a video creation support device 20 according to this embodiment. Fig. 2 shows a video creation support system including the video creation support device 20, a terminal 10, and a generation server 30. In the video creation support system, the multiple terminals 10, the video creation support device 20, and the generation server 30 are configured to be able to communicate with each other via a network including a wireless communication network and a fixed communication network.

[0014] The terminal 10 is a terminal used by a user (viewing user) who views a video. The terminal 10 may be, for example, a personal computer, a smartphone, a tablet terminal, a feature phone, a server device, a game console, or the like. While FIG. 2 illustrates only one terminal 10, the video creation support system actually includes multiple terminals 10. When a video is viewed by a viewing user, the terminal 10 transmits video information (details will be described later) including various information related to the video viewing to the video creation support device 20. The timing for transmitting the video information from the terminal 10 to the video creation support device 20 may be when the viewing user finishes viewing the video, while the viewing user is viewing the video (in real time), or at predetermined time intervals (periodic timing). When video information is transmitted at predetermined time intervals, video information related to all videos viewed between the previous transmission and the current transmission may be transmitted together.

[0015] The generation server 30 is a device that generates videos. The generation server 30 has, for example, a video generation model 31 (generative AI model). The generative AI model is a model that can generate content in response to a prompt including input information, according to any one or a combination of the instructions, context, question, and output format indicated by the prompt, and return the content as response information. The prompt can also include input information, in which case the generative AI model generates response information targeted at the input information. The generative AI model may be, for example, Sora, an open AI video generation model.

[0016] The moving image generation model 31 continuously generates a moving image including a plurality of scenes and transmits the same to the moving image creation support device 20. The moving image generation model 31 also generates a moving image based on feedback (recommendations) from the moving image creation support device 20.

[0017] The moving image creation support device 20 includes, as functional components, a reception unit 21, a model creation unit 22, a prediction unit 23, a recommendation unit 24, and a storage unit 25.

[0018] The reception unit 21 receives video information related to a video viewed by a viewing user from each of the terminals 10. In this way, the reception unit 21 receives multiple pieces of video information. The video information is information related to a video including multiple scenes in chronological order, and includes information about the viewing user, information about each scene, and information about the excitement in each scene.

[0019] The information about the viewing user is various information about the viewing user, who is a user who viewed a video via the terminal 10. The reception unit 21 may receive information about the viewing user, such as the age (or generation), gender, place of residence, hobbies, and preferences. Here, as an example, the description will be given assuming that the reception unit 21 receives the generation and gender of the viewing user as the information about the viewing user.

[0020] The scene information is various information about the scene viewed by the viewing user. The reception unit 21 may receive information about the performers in the scene as the scene information. The performer information is information about the performers who appear in the video if the video is live action, and information about the characters who appear in the video if the video is animation, CG, or the like. The performer information may be information indicating the performer (information such as the performer name that identifies the performer), information indicating the performer's attributes (e.g., young male, etc.), or information indicating the performer's role in the video (e.g., protagonist, partner, rival, parent and child, couple, boss, mentor, best friend, etc.). Note that the performer information can be made more versatile by vectorizing the performer information rather than the performer's name.

[0021] The reception unit 21 may receive a dialogue sentiment analysis result as information about a scene. The dialogue is a line spoken by a character in a video. The dialogue sentiment analysis result is an analysis result of an emotion estimated from transcribed dialogue information (or from the transcribed dialogue information and information about the music played at the same time), and is information indicating, for example, anger, sadness, joy, neutral, etc. The dialogue sentiment analysis may be performed on the terminal 10, on the video creation support device 20, or by another server, etc.

[0022] The receiving unit 21 may receive information about music as information about a scene. The information about music may be information indicating whether music is playing, information indicating the type of music being played (e.g., bright music, dark music, etc.), or information indicating the role of the music being played in the video (e.g., main theme, insert song, etc.).

[0023] The receiving unit 21 may receive information about camera work or camera angle as information about a scene. The information about camera work or camera angle may be information indicating, for example, pan, tilt, zoom in, zoom out, high angle, low angle, horizontal angle, etc.

[0024] The receiving unit 21 may receive, as the information related to the excitement of a scene, at least one of the number of times the corresponding scene is repeated, information related to the volume of the playback of the corresponding scene, and information related to the reaction of the viewing user to the corresponding scene. The receiving unit 21 may also receive, as the information related to the excitement, information other than the above, as long as the information specifies the degree of excitement in the scene.

[0025] The number of times a corresponding scene has been repeated is information indicating how many times the viewing user has repeated the scene. The number of times a corresponding scene has been repeated may be the number of times it has been repeated within a predetermined period (for example, within a predetermined period of several minutes to several days). It is estimated that the greater the number of times a scene has been repeated, the more interested the viewing user is in the scene and the higher the degree of excitement.

[0026] The information about the volume during playback of the corresponding scene may be information indicating the volume itself or information indicating a change in volume over time. The louder the volume, the more interested the viewer is in the scene and the higher the level of excitement is estimated to be. Furthermore, if the volume is gradually increased during the scene, it is estimated that the viewer is interested in the scene and the higher the level of excitement is estimated to be.

[0027] The information regarding the reaction of the viewing user to the corresponding scene is information indicating how the viewing user evaluated the scene. The information regarding the reaction of the viewing user may be, for example, information indicating a qualitative evaluation such as "high rating" or "low rating," or information indicating a quantitative evaluation such as a score. It is estimated that the higher the rating, the more interested the viewing user is in the scene and the higher the degree of excitement.

[0028] The reception unit 21 stores a plurality of pieces of video information received from each terminal 10 in the storage unit 25. The storage unit 25 stores the plurality of pieces of video information input from the reception unit 21. The storage unit 25 also stores a machine learning model created by the model creation unit 22 (described later). The storage unit 25 also stores a prediction result of information related to excitement by the prediction unit 23 (described later).

[0029] The model creation unit 22 creates a machine learning model that predicts information related to excitement by performing machine learning using multiple pieces of video information stored in the storage unit 25. The model creation unit 22 stores the created machine learning model in the storage unit 25.

[0030] FIG. 3 is a diagram illustrating an overview of creating a machine learning model. As shown in FIG. 3, assume that video information for a plurality of videos (here, as an example, a woman in her 30s and a man in his 20s) is stored in the storage unit 25. Each piece of video information is information related to a video including a plurality of scenes, and assumes that the degree of excitement in, for example, six consecutive scenes (scenes from 0 to 10 seconds, 11 to 20 seconds, 21 to 30 seconds, 31 to 40 seconds, 41 to 50 seconds, and 51 to 60 seconds) is indicated by a score. Using this video information, the model creation unit 22 machine-learns information indicating which viewing users (here, a woman in her 30s or a man in his 20s) were excited by which scenes, and to what extent, thereby creating a machine learning model that predicts information about the excitement level for a target user.

[0031] FIG. 4 is a diagram illustrating the details of creating a machine learning model. FIG. 4 shows an example of detailed video information for a certain viewing user (a man in his twenties). Here, the video information shows scores representing the degree of excitement in six consecutive scenes (scenes from 0 to 10 seconds, 11 to 20 seconds, 21 to 30 seconds, 31 to 40 seconds, 41 to 50 seconds, and 51 to 60 seconds). These scores may be normalized to a value between 0 and 100, for example. These scores may be determined from statistical values ​​for the same attribute (a man in his twenties) or may be values ​​for a specific individual. Information about each scene also includes information about the cast, sentiment analysis results for the dialogue, and information about the music. For example, for the "0-10 second scene," the cast's "protagonist" and "partner" are "True," "rival" is "False" (the protagonist and partner appear, the rival does not appear), the sentiment analysis results for the lines show that "sadness" is "True" and "joy" is "False" (the lines are related to sadness, not joy), the music's "theme song" is "False," and "insert song" is "False" (the music is not the theme song, not an insert song).

[0032] The model creation unit 22 performs machine learning on multiple pieces of video information, using information about the viewing user and information about each scene as explanatory variables and information about the excitement level of each scene as a target variable, to create a machine learning model that predicts information about the excitement level. For example, in the example of the "0-10 second scene" of video information shown in FIG. 4, information about the viewing user (a man in his 20s) and information about the scene (the cast members are the protagonist and his partner, the sentiment analysis result of the dialogue is sad, and no theme song or insert song is played) are used as explanatory variables, and a score of "80" indicating the level of excitement is used as the target variable. This process is performed for each scene of each piece of video information, and machine learning is performed to create a machine learning model that predicts information about the excitement level. Machine learning may be performed using an algorithm such as a random forest or a recurrent neural network. The machine learning model predicts information about the excitement level by inputting information about the user and information about the scene. Information about each scene may be characterized, and a machine learning model that semantically characterizes image features such as CLIP may be used for the characterization.

[0033] The model creation unit 22 may create a machine learning model by performing machine learning using only scenes of a plurality of video information whose excitement level (score) indicated by the excitement information is equal to or greater than a predetermined value. For example, when using the video information shown in Figure 4, if it is predetermined to use only scenes whose excitement level (score) is equal to or greater than 50, the "scene from 0 to 10 seconds" (score 80), the "scene from 31 to 40 seconds" (score 50), and the "scene from 51 to 60 seconds" (score 100) may be used as learning data, and other scenes may be excluded from the learning data.

[0034] The model creation unit 22 may create a machine learning model by learning not only information about the excitement of a single scene, but also information about changes in the information about the excitement over time. For example, the model creation unit 22 may perform machine learning using information about the excitement of each scene and information about the excitement of the scene following each scene as objective variables, to create a machine learning model that performs a first inference that predicts information about the excitement of each scene and a second inference that predicts information about the excitement of the scene following each scene. For example, in the example shown in FIG. 4 , machine learning may be performed using information about the excitement of successive scenes as objective variables, such that "40," which is the excitement level (score) of the "scene from 21 to 30 seconds," is set as "target variable 1," and "50," which is the excitement level (score) of the subsequent "scene from 31 to 40 seconds," is set as "target variable 2."

[0035] The prediction unit 23 predicts information regarding the excitement in each target scene by inputting multiple pieces of information regarding the target user and information regarding the target scene for which excitement is to be predicted into a machine learning model stored in the memory unit 25.

[0036] Information about the target scene whose excitement is predicted is received, for example, from the generation server 30. In this case, the generation server 30 continuously generates videos including multiple scenes (target scenes) using the video generation model 31 and transmits them to the video creation support device 20. The prediction unit 23 acquires information about the target scene whose excitement is predicted based on the video (video including multiple target scenes) transmitted from the generation server 30. The information about the target scene may be the same as the information about the scene received by the above-mentioned reception unit 21, and may be, for example, information about the cast, sentiment analysis results of the dialogue, information about the music, information about camerawork or camera angles, etc., related to the target scene. The information about the target user may, for example, be determined in advance, or may be received each time together with the video from the generation server 30.

[0037] 5 is a diagram illustrating details of excitement prediction and video recommendation. As shown in FIG. 5, excitement prediction and video recommendation may be performed by continuously generating videos using a video generation model 31 of a generation server 30. With this configuration, excitement evaluation is performed on each target scene of the generated video, and the video is repeatedly regenerated until the evaluation improves, making it possible to create a video including scenes with high excitement evaluations.

[0038] The "movie composition evaluator" shown in FIG. 5 is a machine learning model that can perform the first and second inferences described above. As shown in FIG. 5, first, a "movie 1" is generated by the movie generation model 31. "Movie 1" is configured to include the first scene included in the target scene described above. Here, the description will be given assuming that "movie 1" includes only the first scene. Then, by inputting information about the first scene into the movie composition evaluator (machine learning model), a second inference is performed to predict information about the excitement in the scene following the first scene.

[0039] Next, the moving image generation model 31 generates "a part (second half) of moving image 1 and a continuous moving image 2." "Movie 2" is configured to include a second scene, which is a candidate for the next scene after the first scene described above. Here, the description will be given assuming that "movie 2" includes only the second scene. Then, information about the part of the first scene and the second scene is input to the moving image composition evaluator (machine learning model), and a first inference is performed to predict information about the excitement in the second scene.

[0040] Then, the video is repeatedly regenerated using the video generation model 31 until the result of the first inference approaches the result of the second inference (within a predetermined error range). In this case, the input parameters related to video generation may be comprehensively changed. In this case, if there are multiple values ​​of explanatory variables that can bring the result of the second inference closer to the result of the second inference, such explanatory variable values ​​may be automatically selected or may be selected by the user. By determining the first scene and the scene following the first scene through such processing, a video composition preferred by the target user can be created.

[0041] As described above, the prediction unit 23 performs a first process of predicting information about the excitement of the scene following the first scene by inputting information about a first scene included in a plurality of target scenes into a machine learning model and performing a second inference. Also, the prediction unit 23 performs a second process of predicting information about the excitement of the second scene by inputting information about a second scene that is a candidate for the scene following the first scene into a machine learning model and performing a first inference.

[0042] If the error between the prediction result of the information related to excitement in the first process and the prediction result of the information related to excitement in the second process is within a predetermined range, the prediction unit 23 determines the second scene as the scene following the first scene (third process). On the other hand, if the error is not within the predetermined range, the prediction unit 23 sets a new second scene with changed parameters in order to perform the second process described above again (third process). The prediction unit 23 instructs, for example, the animation generation model 31 to change the parameters. As a result, the animation generation model 31 generates a new second scene. The prediction unit 23 performs first inference using the new second scene (second process) and performs the third process again. The prediction unit 23 repeats the third process and the second process until the error between the prediction result of the information related to excitement in the first process and the prediction result of the information related to excitement in the second process is within a predetermined range. The prediction unit 23 stores the prediction result of the information related to excitement in the storage unit 25.

[0043] The recommendation unit 24 recommends a video including one or more target scenes based on the prediction result by the prediction unit 23. The recommendation unit 24 identifies one or more target scenes that will excite the target user by referring to the storage unit 25, and recommends a video including the target scenes. The recommendation by the recommendation unit 24 is transmitted to, for example, the generation server 30. Then, the generation server 30 generates a video according to the recommendation using the video generation model 31. The recommendation unit 24 may recommend a video including the above-mentioned first scene and a second scene determined to be the next scene after the first scene by referring to the storage unit 25.

[0044] Next, the process executed by the moving image creation support device 20 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the process executed by the moving image creation support device 20.

[0045] As shown in FIG. 6, first, the moving image creation support device 20 receives moving image information (plural pieces of moving image information) from each terminal 10 (step S1).

[0046] Next, in the video creation support device 20, machine learning is performed on multiple video information using information about the viewing user and information about each scene as explanatory variables and information about the excitement in each scene as the objective variable, and a machine learning model that predicts information about the excitement is created (step S2).

[0047] Next, in the video creation support device 20, multiple pieces of information about the target user and information about the target scenes for which excitement is to be predicted are input into the machine learning model, and information about the excitement in each target scene is predicted (step S3).

[0048] Finally, the video creation support device 20 recommends videos that include one or more target scenes based on the prediction results regarding the excitement in each target scene described above (step S4).

[0049] Next, the effects of the moving image creation support device 20 according to this embodiment will be described.

[0050] The video creation support device 20 comprises a reception unit 21 that receives multiple pieces of video information related to a video including multiple scenes in chronological order, the video information including information about the viewing user, information about each scene, and information about the excitement in each scene; a model creation unit 22 that performs machine learning on the multiple pieces of video information using information about the viewing user and information about each scene as explanatory variables and information about the excitement in each scene as a target variable, and creates a machine learning model that predicts information about the excitement; a prediction unit 23 that predicts information about the excitement in each target scene by inputting multiple pieces of information about the target user and information about the target scene for which the excitement is to be predicted into the machine learning model; and a recommendation unit 24 that recommends videos that include one or more target scenes based on the prediction results by the prediction unit 23.

[0051] In the video creation support device 20, a machine learning model is created using information about the viewing user and information about each scene as explanatory variables and information about the excitement in each scene as a target variable. Then, multiple pieces of information about the target user and information about the target scenes are input into the machine learning model, and information about the excitement in each target scene is predicted. Based on the prediction results, videos containing one or more target scenes are recommended. With this configuration, a machine learning model trained by associating information about the viewing user and the scenes with information about the excitement can be used to accurately predict information about the excitement in each target scene for the target user. By recommending videos containing one or more target scenes based on these prediction results, videos that are likely to excite the target user can be easily and appropriately recommended. As described above, the video creation support device 20 according to this embodiment can create videos that users can enjoy in a short period of time.

[0052] Conventionally, there have been problems such as not knowing what kind of videos will be popular, not knowing what aspects of popular videos are popular, and difficulty in recommending videos because each user has different preferences. In response to these problems, the configuration according to the present embodiment described above acquires scenes that are exciting for each user from the video and uses machine learning to learn information about the scenes, making it possible to appropriately and easily create exciting videos when creating new videos.

[0053] The receiving unit 21 may receive information about performers as information about scenes. By receiving information about performers who have a large influence on the excitement of a video and using it as learning data, the prediction accuracy of the information about the excitement by the machine learning model can be improved.

[0054] The reception unit 21 may receive a sentiment analysis result of a line as information related to a scene. By receiving the sentiment analysis result of a line that has a large influence on the excitement of a video and using it as training data, it is possible to improve the prediction accuracy of information related to excitement by the machine learning model.

[0055] The receiving unit 21 may receive information about music as information about a scene. By receiving information about music that has a large influence on excitement in a video and using it as training data, it is possible to improve the prediction accuracy of information about excitement by the machine learning model.

[0056] The receiving unit 21 may receive information about camera work or camera angles as information about a scene. By receiving information about camera work or camera angles that have a large influence on excitement in a video and using it as learning data, it is possible to improve the prediction accuracy of information about excitement by the machine learning model.

[0057] The receiving unit 21 may receive, as the information related to excitement, at least one of the number of times the corresponding scene is repeated, information related to the volume of the corresponding scene when it is played, and information related to the reaction of the viewing user to the corresponding scene. With this configuration, it is possible to appropriately identify the level of interest (level of excitement) of the user.

[0058] The model creation unit 22 may perform machine learning using only scenes from a plurality of video information whose excitement level indicated by the excitement information is equal to or greater than a predetermined value, thereby creating a machine learning model. This allows the necessary machine learning to be performed appropriately while reducing the learning load.

[0059] The model creation unit 22 performs machine learning using information about the excitement in each scene and information about the excitement in the scene following each scene as objective variables, to create a machine learning model that performs a first inference to predict information about the excitement in each scene and a second inference to predict information about the excitement in the scene following each scene, and the prediction unit 23 inputs information about a first scene included in a plurality of target scenes into the machine learning model and performs a second inference, thereby performing a first process to predict information about the excitement in the scene following the first scene and information about a second scene that is a candidate for the scene following the first scene. a second process of predicting information related to excitement in the second scene by inputting the above into a machine learning model and performing a first inference, and a third process of determining the second scene as the scene following the first scene if the error between the predicted result of the information related to excitement in the first process and the predicted result of the information related to excitement in the second process is within a predetermined range, and setting a new second scene with changed parameters to perform the second process again if the error is not within the predetermined range, and the recommendation unit 24 may recommend a video including the first scene and the second scene determined to be the scene following the first scene. With this configuration, it is possible to appropriately create a video that includes consecutive scenes and has a high level of excitement.

[0060] The present disclosure is not limited to the above-described embodiments. For example, the target video may be a video in a metaverse space. In the metaverse space, each user has a different viewpoint, which affects the level of excitement. Information that affects the level of excitement includes information within the user's field of view. Information within the user's field of view includes, for example, the size of the screen occupied by a character on stage, the number of eye contacts with the character on stage, and the way the light hits the user. In this case, the reception unit 21 receives, as scene-related information, information within the viewing user's field of view related to the corresponding scene. By creating a machine learning model that predicts information regarding excitement using such scene-related information as explanatory variables, it is possible to create videos in the metaverse space that users can enjoy in a short period of time.

[0061] The moving image creation support device and moving image creation support method according to the present disclosure have the following configuration.

[0062] [1] A video creation support device comprising: a reception unit that receives a plurality of video information items related to a video including a plurality of scenes arranged in time series, the video information including information about the viewing user, information about each scene, and information about the excitement in each scene; a model creation unit that performs machine learning on the plurality of video information items using information about the viewing user and information about each scene as explanatory variables and information about the excitement in each scene as a target variable, and creates a machine learning model that predicts information about the excitement; a prediction unit that predicts information about the excitement in each target scene by inputting a plurality of pieces of information about a target user and information about a target scene for which excitement is to be predicted into the machine learning model; and a recommendation unit that recommends a video that includes one or more of the target scenes based on the prediction result by the prediction unit.

[0063] [2] The moving image creation support device according to [1], wherein the reception unit receives information about performers as information about the scene.

[0064] [3] The video creation support device according to [1] or [2], wherein the reception unit receives a sentiment analysis result of dialogue as information about the scene.

[0065] [4] The moving image creation support device according to any one of [1] to [3], wherein the reception unit receives information about music as the information about the scene.

[0066] [5] The moving image creation support device according to any one of [1] to [4], wherein the reception unit receives information about camera work or camera angle as the information about the scene.

[0067] [6] The video is a video in a metaverse space,

[0068] The video creation support device according to any one of [1] to [5], wherein the reception unit receives, as the information about the scene, information about a corresponding scene within the field of view of a viewing user.

[0069] [7] The video creation support device according to any one of [1] to [6], wherein the reception unit receives, as the information relating to the excitement, at least one of the number of times the corresponding scene is repeated, information relating to the volume of the playback of the corresponding scene, and information relating to the reaction of the viewing user to the corresponding scene.

[0070] [8] The video creation support device according to any one of [1] to [7], wherein the model creation unit performs machine learning using only scenes among the plurality of video information whose excitement level indicated by the excitement information is equal to or greater than a predetermined value, to create the machine learning model.

[0071] [9] The model creation unit performs machine learning using information about excitement in each of the scenes and information about excitement in a scene following each of the scenes as objective variables to create the machine learning model that performs a first inference to predict information about excitement in each of the scenes and a second inference to predict information about excitement in a scene following each of the scenes, and the prediction unit performs: a first process to predict information about excitement in a scene following the first scene by inputting information about a first scene included in a plurality of the target scenes into the machine learning model and performing the second inference; a second process to predict information about excitement in the second scene by inputting information about a second scene that is a candidate for the first scene into the machine learning model and performing the first inference; and a third process to determine the second scene as the scene following the first scene when an error between a prediction result of information about excitement in the first process and a prediction result of information about excitement in the second process is within a predetermined range, and to set a new second scene with changed parameters in order to perform the second process again when an error is not within the predetermined range. The video creation support device according to any one of [1] to [8], wherein the recommendation unit recommends a video comprising the first scene and the second scene determined to be the next scene after the first scene.

[0072]

[10] A video creation support method performed by a video creation support device, the video creation support method comprising: receiving a plurality of video information items relating to a video including a plurality of scenes in chronological order, the video information items including information about a viewing user, information about each scene, and information about excitement in each scene; performing machine learning on the plurality of video information items using the information about the viewing user and the information about each scene as explanatory variables and the information about excitement in each scene as a target variable, to create a machine learning model that predicts information about excitement; predicting information about excitement in each target scene by inputting a plurality of pieces of information about a target user and information about a target scene for which excitement is predicted into the machine learning model; and recommending a video that includes one or more of the target scenes based on the prediction results.

[0073] The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of hardware and / or software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using a single device that is physically or logically coupled, or may be realized using two or more physically or logically separated devices that are connected directly or indirectly (e.g., via wire, wirelessly, etc.) and these multiple devices. The functional block may also be realized by combining the single device or multiple devices with software.

[0074] Functions include, but are not limited to, judgment, determination, assessment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.

[0075] For example, a video creation support device 20 constituting a video creation support system according to an embodiment of the present disclosure may function as a computer that performs processing of the control method of the present disclosure. FIG. 7 is a diagram illustrating an example of the hardware configuration of the video creation support device 20 according to this embodiment. The video creation support device 20 described above may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like. Note that the video creation support device 20 may be configured as a computer device including at least one processor such as a CPU or GPU, or may be configured as a computer device including multiple processors or may be configured to include multiple computer devices. The terminal 10 and the generation server 30 may also have a similar hardware configuration.

[0076] In the following description, the term "apparatus" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the video creation support apparatus 20 may be configured to include one or more of the devices shown in the figure, or may be configured to exclude some of the devices.

[0077] Each function in the video creation support device 20 is realized by loading specified software (programs) onto hardware such as the processor 1001 and memory 1002, causing the processor 1001 to perform calculations, control communication via the communication device 1004, and control at least one of reading and writing data in the memory 1002 and storage 1003.

[0078] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured by a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the above-mentioned reception unit 21, model creation unit 22, prediction unit 23, recommendation unit 24, etc. may be realized by the processor 1001.

[0079] The processor 1001 also reads programs (program code), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002 and executes various processes in accordance with these. The program used may be a program that causes a computer to execute at least some of the operations described in the above-described embodiments. For example, the reception unit 21, the model creation unit 22, the prediction unit 23, and the recommendation unit 24 may be implemented by a control program stored in the memory 1002 and running on the processor 1001, and similar implementations may be used for other functional blocks. While the above-described various processes have been described as being executed by one processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may also be transmitted from a network via a telecommunications line.

[0080] The memory 1002 is a computer-readable recording medium and may be configured, for example, by at least one of a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a random access memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing a control method according to an embodiment of the present disclosure.

[0081] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other appropriate medium including at least one of memory 1002 and storage 1003.

[0082] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 1004 may be configured to include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc. to realize at least one of frequency division duplex (FDD) and time division duplex (TDD). For example, the above-mentioned reception unit 21, recommendation unit 24, etc. may be realized by the communication device 1004.

[0083] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that outputs to the outside. Note that the input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0084] Furthermore, each device, such as the processor 1001 and the memory 1002, is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0085] The video creation support device 20 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0086] The notification of information is not limited to the aspects / embodiments described in the present disclosure and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI) and Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB) and System Information Block (SIB))), other signals, or a combination thereof. Furthermore, the RRC signaling may be referred to as an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.

[0087] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0088] Input and output information may be stored in a specific location (for example, memory) or may be managed using a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.

[0089] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0090] The aspects / embodiments described in this disclosure may be used alone, in combination, or switched depending on the implementation. Notification of predetermined information (e.g., notification that "X is true") is not limited to explicit notification, but may be implicit (e.g., not notifying the predetermined information).

[0091] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0092] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0093] Software, instructions, information, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave), then these wired and / or wireless technologies are included within the definition of transmission media.

[0094] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0095] Note that terms described in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings. For example, at least one of a channel and a symbol may be a signal (signaling). Furthermore, a signal may be a message. Furthermore, a component carrier (CC) may be called a carrier frequency, a cell, a frequency carrier, etc.

[0096] Furthermore, the information, parameters, etc. described in the present disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or other corresponding information. For example, a radio resource may be indicated by an index.

[0097] The names used for the above-described parameters are not intended to be limiting in any way. Furthermore, the mathematical expressions using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not intended to be limiting in any way.

[0098] In this disclosure, the terms "Mobile Station (MS)," "user terminal," "User Equipment (UE)," "terminal," and the like may be used interchangeably.

[0099] A mobile station may also be referred to by those skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology.

[0100] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0101] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using one or more wires, cables, and / or printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0102] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0103] As used in this disclosure, any reference to an element using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must in some way precede the second element.

[0104] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, when the term "or" is used in this disclosure, it is not intended to be an exclusive or.

[0105] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0106] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0107] 20...video creation support device, 21...reception unit, 22...model creation unit, 23...prediction unit, 24...recommendation unit.

Claims

1. A video creation support device comprising: a reception unit that receives multiple pieces of video information related to a video including multiple scenes in chronological order, the video information including information about the viewing user, information about each scene, and information about the excitement in each scene; a model creation unit that performs machine learning on the multiple pieces of video information using information about the viewing user and information about each scene as explanatory variables and information about the excitement in each scene as a target variable, and creates a machine learning model that predicts information about the excitement; a prediction unit that predicts information about the excitement in each target scene by inputting multiple pieces of information about the target user and information about the target scene for which the excitement is to be predicted into the machine learning model; and a recommendation unit that recommends videos that include one or more of the target scenes based on the prediction results by the prediction unit.

2. The moving image creation support device according to claim 1, wherein said reception unit receives information about performers as information about said scenes.

3. The moving image creation support device according to claim 1, wherein the receiving unit receives a result of a sentiment analysis of dialogue as information about the scene.

4. The moving image creation support device according to claim 1, wherein said reception unit receives information about music as the information about the scene.

5. The video creation support device according to claim 1, wherein the reception unit receives information about camera work or camera angles as the information about the scene.

6. The video creation support device according to claim 1, wherein the video is a video in a metaverse space, and the reception unit receives information about the corresponding scene within the viewing user's field of view as information about the scene.

7. The video creation support device of claim 1, wherein the reception unit receives as the information regarding the excitement at least one of the number of times the corresponding scene is repeated, information regarding the volume of the playback of the corresponding scene, and information regarding the reaction of the viewing user to the corresponding scene.

8. The video creation support device of claim 1, wherein the model creation unit performs machine learning using only scenes from the plurality of video information whose excitement level indicated by the excitement information is equal to or greater than a predetermined value, to create the machine learning model.

9. The model creation unit performs machine learning using information about excitement in each of the scenes and information about excitement in a scene following each of the scenes as objective variables, to create the machine learning model that performs a first inference to predict information about excitement in each of the scenes and a second inference to predict information about excitement in a scene following each of the scenes; the prediction unit performs: a first process to predict information about excitement in a scene following the first scene by inputting information about a first scene included in a plurality of the target scenes into the machine learning model and performing the second inference; a second process to predict information about excitement in a scene following the first scene by inputting information about a second scene that is a candidate for the scene following the first scene into the machine learning model and performing the first inference; and a third process to determine the second scene as the scene following the first scene if an error between a prediction result of information about excitement in the first process and a prediction result of information about excitement in the second process is within a predetermined range, and to set a new second scene with changed parameters in order to perform the second process again if an error is not within the predetermined range; The video creation support device according to any one of claims 1 to 8, wherein the recommendation unit recommends a video comprising the first scene and the second scene determined to be the next scene after the first scene.

10. A video creation support method performed by a video creation support device, comprising: receiving a plurality of pieces of video information relating to a video including a plurality of scenes in chronological order, the video information including information about the viewing user, information about each scene, and information about the excitement in each scene; performing machine learning on the plurality of pieces of video information using the information about the viewing user and the information about each scene as explanatory variables and the information about the excitement in each scene as a target variable, to create a machine learning model that predicts information about the excitement; predicting information about the excitement in each target scene by inputting a plurality of pieces of information about the target user and information about target scenes for which excitement is predicted into the machine learning model; and recommending a video that includes one or more of the target scenes based on the prediction results.