Information processing device, information processing method, and program

Abstract

An information processing device for acquiring content of interest having content that changes with time and predetermined time-series information recorded while the content of interest is outputted, acquiring model output data corresponding to element data constituting the content of interest by using a machine learning model obtained by machine learning a relationship between element data included in the content and description information linguistically describing the content of the element data, and executing predetermined processing on the basis of the acquired time-series information and the acquired model output data.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program for analyzing moving images.

[0002] When a user plays a video game, the information processing device that executes the program renders moving images whose content changes depending on the progress of the game, and presents them to the user. By analyzing the moving images described above, it is possible to obtain information such as how the user is playing in a particular scene. In this way, it is desirable to analyze the content of various content presented to the user and obtain useful information.

[0003] However, the content that users generally view is often massive in both playback time and data volume, making manual analysis impractical. Furthermore, when analysis is performed mechanically by a computer rather than manually, the information obtained is not necessarily meaningful to humans.

[0004] The present invention has been made in consideration of the above-mentioned circumstances, and one of its objectives is to provide an information processing device, an information processing method, and a program that can effectively analyze content presented to a user.

[0005] An information processing device according to one aspect of the present invention is an information processing device having one or more processors, which acquire target content having content that changes over time and predetermined time series information recorded while the target content is being output, acquire model output data corresponding to the element data that constitutes the target content using a machine learning model obtained by machine learning the relationship between element data included in the content and descriptive information that linguistically describes the content of the element data, and perform predetermined processing based on the acquired time series information and the acquired model output data.

[0006] An information processing method according to one aspect of the present invention is an information processing method that acquires target content having content that changes over time and predetermined time series information recorded while the target content is being output, acquires model output data corresponding to the element data that constitutes the target content using a machine learning model obtained by machine learning the relationship between element data included in the content and descriptive information that linguistically describes the content of the element data, and performs predetermined processing based on the acquired time series information and the acquired model output data.

[0007] A program according to one aspect of the present invention is a program for causing a computer to execute a process of acquiring target content having content that changes over time and predetermined time-series information recorded while the target content is being output, acquiring model output data corresponding to the element data constituting the target content using a machine learning model obtained by machine learning the relationship between element data included in the content and descriptive information that linguistically describes the content of the element data, and executing predetermined processing based on the acquired time-series information and the acquired model output data. This program may be provided by being stored on a computer-readable, non-transitory information storage medium.

[0008] It is a configuration block diagram showing the configuration of an information processing device according to an embodiment of the present invention. It is a functional block diagram showing the functions of the information processing device according to an embodiment of the present invention. It is a diagram for explaining an example of a learning process for generating a machine learning model used by the information processing device according to an embodiment of the present invention. It is a diagram for explaining the relationship between model output data corresponding to element data constituting target content and time-series information.

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0010] 1 is a block diagram showing the configuration of an information processing device 10 according to one embodiment of the present invention. The information processing device 10 is a personal computer, a server computer, or the like, and as shown in the figure, includes a control unit 11, a storage unit 12, and an interface unit 13. The information processing device 10 is also connected to a display device 14 and an operation device 15.

[0011] The control unit 11 includes at least one processor such as a CPU, and performs various information processing by executing programs stored in the storage unit 12. Specific examples of the processing performed by the control unit 11 in this embodiment will be described later. The storage unit 12 includes at least one memory device such as a RAM, and stores the programs executed by the control unit 11 and data processed by the programs.

[0012] The interface unit 13 is an interface for data communication between the display device 14 and the operation device 15. The information processing device 10 is connected to the display device 14 and the operation device 15 via the interface unit 13 either wired or wirelessly. Specifically, the interface unit 13 includes a multimedia interface for transmitting a video signal supplied by the information processing device 10 to the display device 14. It also includes a data communication interface for receiving a signal indicating the operation content performed by the user on the operation device 15.

[0013] The display device 14 displays on its screen an image corresponding to a video signal supplied from the information processing device 10. The operation device 15 is, for example, a keyboard or a mouse, and receives operation input from a user. The operation device 15 is connected to the information processing device 10 by wire or wirelessly, and transmits an operation signal indicating the content of the operation input received from the user to the information processing device 10.

[0014] The functions realized by the information processing device 10 will be described below with reference to the functional block diagram of Fig. 2. As shown in Fig. 2, the information processing device 10 is functionally configured to include a target content acquisition unit 21, a model output data acquisition unit 22, and an analysis unit 23. These functions are realized by the control unit 11 operating in accordance with one or more programs stored in the storage unit 12. These programs may be provided to the information processing device 10 via a communication network such as the Internet, or may be provided by being stored on a computer-readable information storage medium such as an optical disc.

[0015] The target content acquisition unit 21 acquires data of content to be analyzed. Hereinafter, the content to be analyzed by the information processing device 10 according to this embodiment will be referred to as target content C. This target content C is content whose content changes over time, and is typically video content including moving images and audio. Hereinafter, target content C is assumed to be content that has actually been played and presented to a user. Furthermore, a user who viewed target content C when it was output will be referred to as user U. However, target content C is not limited to this, and may be various types of content that change over time and are output over a certain period of time, such as content consisting of only moving images or only audio.

[0016] As an example, the target content C may be a game image that is drawn by the user terminal and displayed on a display device to present to the user U when the user U plays the game using the user terminal that the user terminal owns. In this case, the target content C is an image that changes in real time according to the operation input made by the user U to an operation device connected to the user terminal.

[0017] Furthermore, the target content acquisition unit 21 acquires, together with the target content C, predetermined time-series information S recorded while the target content C is being output and presented to the user U.

[0018] As a specific example, the time-series information S may be time-series operation information indicating the content of operation inputs made to an operation device by a user U watching target content C, which is a game video, while the target content C is being output. The time-series operation information includes information that can identify the content of the operation made by the user U (for example, a pressed button, or the direction and amount of operation of a stick-type operation member) and the timing of the operation.

[0019] Furthermore, the time-series information S is not limited to time-series operation information, and may be various types of information related to the status of the user U while the target content C is being output. For example, the time-series information S may include information obtained by recording in real time, using various sensors, vital signs (such as heart rate) and behavior (such as body movements) of the user U while watching the target content C. The time-series information S may also include information such as the user U's facial expressions and eye movements obtained by capturing an image of the user U with a camera. The target content acquisition unit 21 may also acquire, as the time-series information S, information obtained by analyzing such information recorded in real time. For example, by analyzing an image captured of the user U or information related to the user U's body movements, the target content acquisition unit 21 may acquire, as the time-series information S, information indicating that the user U left their seat at a predetermined timing, information indicating the emotions the user U felt at each point in time, information regarding the degree to which the user U was concentrating on the target content C, information indicating which part of the target content C the user was paying attention to, and the like.

[0020] By combining and referencing this target content C and the time series information S, the information processing device 10 can identify the content of the operation performed by the user U at the time a certain image is displayed, as well as the situation of the user U at that time, such as the facial expression, posture, and emotions of the user U.

[0021] The model output data acquisition unit 22 uses a machine learning model M prepared in advance to acquire model output data D corresponding to each of one or more elements constituting the target content C. This machine learning model M is a machine learning model obtained by learning the relationship between element data included in the content and descriptive information used to linguistically describe the content of the element. For ease of explanation, the machine learning model M is assumed to be a model obtained by learning the relationship between various types of image data and descriptive information that linguistically describes the content of the image. However, without being limited to this, the model output data acquisition unit 22 may also use a machine learning model M obtained by learning the relationship between elements constituting the target content C, such as short sounds that are played during playback of the target content C, and descriptive information that linguistically describes the content of the image.

[0022] The machine learning model M used in this embodiment may be a model based on various techniques as long as it can acquire information used to linguistically describe the content of the elements that make up the target content C. For example, if the analysis target is image data showing a horse galloping with a man on it, it is desirable that this machine learning model M not simply output the names and positions of objects (horse, man, etc.) contained in the image, but rather a model that ultimately obtains a descriptive sentence such as "A samurai is galloping on horseback through the wilderness." A machine learning model M capable of such output can be generated, for example, by a technique such as CLIP (Contrastive Language-Image Pre-training).

[0023] However, in this embodiment, the model output data D acquired by the model output data acquisition unit 22 as an output from the machine learning model M does not need to be explanatory sentences that humans can understand, but may be information corresponding to linguistic information. An example of a learning process for generating the machine learning model M used by the information processing device 10 according to this embodiment will be described below with reference to FIG. 3.

[0024] The learning processing device that executes the learning process in this example accepts multiple sets of training data, each set consisting of image data and description information that linguistically describes the image data. For each set of training data, the learning device divides the description information contained in a given set into tokens (e.g., words) t and obtains its distributed representation tref (token embedding). The learning processing device then generates description-related data ft(tref) by encoding the tokens using a text encoder. This description-related data ft(tref) is multidimensional vector data and corresponds to one piece of coordinate information in an abstract space.

[0025] The learning processing device also generates image-related data fi(I) by inputting image data I included in the same set as the description information T into an image encoder. The learning processing device sets the image encoder so that this image-related data fi(I) also becomes vector data of the same dimension as the description-related data ft(tref). This image-related data fi(I) also corresponds to one piece of coordinate information in the abstract space.

[0026] The learning processing device performs machine learning on each encoder ft, fi so that the description-related data ft(tref) and the image-related data fi(I) are arranged in close proximity to each other in the predetermined abstract space (a vector-dimensional space of the description-related data and the image-related data, known as a multimodal embedding space in CLIP). Conceptually, this results in the image data and the description information describing it being associated with each other in the predetermined abstract space P (S11), as shown in the example of Figure 3. In the above example, the text encoder ft may use a well-known transformer network, and the image encoder fi may use a Vision Transformer (ViT).

[0027] In this embodiment, data of the machine learning model M generated by such machine learning is stored in advance in the storage unit 12. The model output data acquisition unit 22 inputs data of elements constituting the target content C (e.g., frame images constituting the moving image of the target content C) acquired by the target content acquisition unit 21 to the machine learning model M, and outputs the output as model output data D.

[0028] When using the machine learning model M that converts each element into a vector in the abstract space P described above, the model output data D may be the image-related data fi(I) itself obtained by inputting frame image data I that constitutes the target content C into an image encoder fi. Alternatively, the model output data D may be data relating to a character string representation obtained by inputting this image-related data fi(I) into a decoder that performs the reverse conversion to a text encoder that converts tokens corresponding to descriptive information T into a vector representation. Such model output data D becomes data that more directly represents words and sentence representations related to the frame image data I.

[0029] Alternatively, an index value (such as cosine similarity) indicating the distance in the abstract space P between the description-related data ft(tref) and the image-related data fi(I) corresponding to previously prepared description information T may be calculated, and the evaluation result using this index value may be used as the model output data D. By using such an index value, it is possible to evaluate the extent to which the frame image data I corresponds to the content expressed by the predetermined description information T. For example, by preparing multiple pieces of description information T related to violent expressions in advance and evaluating the degree to which the frame image data I to be analyzed is close to these pieces of description information T in the abstract space P, it is possible to determine whether the frame image data I contains violent expressions.

[0030] Furthermore, the model output data D may be data obtained by performing clustering on the distribution within an abstract space P obtained by converting multiple image data into vector representations. For example, a clustering process is performed in which multiple image-related data fi(I) obtained by inputting multiple image data extracted in a predetermined manner into an image encoder fi is grouped within the abstract space P. If, for example, three clusters are extracted as a result, it can be estimated that the image data belonging to each of these clusters are images with similar meanings. Furthermore, description information T corresponding to the position of each cluster may be obtained as the model output data D. This description information T can be used to roughly explain the content of the image data belonging to the corresponding cluster.

[0031] Here, the description information T corresponding to the position of the cluster may be the description information T corresponding to the vector that is the shortest distance from the center position of the cluster among vectors obtained by converting multiple candidate description information T as references prepared in advance. Alternatively, the description information T corresponding to the center position of the cluster may be obtained using a decoder that performs the inverse conversion of the text encoder described above. Such a decoder can be generated by machine learning, which performs tasks such as converting character strings into vectors and then restoring them to the original character strings for a prepared corpus. Alternatively, the description information T obtained as a sentence expression may be input again into a predetermined encoder to obtain an embedding, which may be used as the model output data D. The character strings corresponding to the multiple clusters obtained in this manner can be visualized using a technique such as a word cloud. When extracting clusters from the vector distribution in the abstract space P, clusters may be extracted from the entire distribution in the abstract space P obtained by inputting image data obtained from the entire target content C, or a portion of vectors that satisfy specific conditions may be extracted as clusters. In either case, the model output data D obtained by inputting certain image data into the machine learning model M is data that can be used to represent the content of the image data in a manner that is understandable to humans. The analysis unit 23, which will be described later, analyzes the details of the target content C using this model output data D.

[0032] Note that the machine learning model M described above is merely an example, and in this embodiment, the model output data acquisition unit 22 can use various models that can obtain the meaning of an image or descriptive information about the image as the machine learning model M. For example, the model output data acquisition unit 22 may use, as the machine learning model M, a model that has been trained to obtain answers to questions about an image by using images, questions about the images, and the answers to those questions as learning objects. Furthermore, a model that has been trained to extract information about a specific perspective by inputting sentences or the like that condition the perspective of information extraction along with the image may be used as the machine learning model M. By using such a machine learning model, the analysis unit 23, which will be described later, can analyze the target content C focusing on a specific perspective.

[0033] Furthermore, in the explanation so far, the machine learning model M accepts as input frame image data constituting the target content C, but it is also possible to use a machine learning model M that accepts as input relatively short videos. This makes it possible to perform analysis that takes into account the movement of objects included in the videos.

[0034] Here, the image data from which the model output data acquisition unit 22 acquires the model output data D may be data on a portion of elements selected according to a predetermined criterion from among the elements that make up the target content C. Specifically, for example, the model output data acquisition unit 22 extracts frame images presented to the user at intervals of, for example, one second, and uses these as the target for acquiring the model output data D. Alternatively, the model output data acquisition unit 22 may acquire the model output data D from frame images that are displayed at the time when data that satisfies a specific condition appears in the time-series information S (for example, the time when the user performs a predetermined operation). A specific example of selecting elements based on the content of the time-series information S will be described later.

[0035] The analysis unit 23 analyzes the target content C by executing a predetermined process based on the time-series information S acquired by the target content acquisition unit 21 together with the target content C and the model output data D acquired by the model output data acquisition unit 22 and corresponding to the element data constituting the target content C. The analysis unit 23 may perform analysis from various perspectives, but in this embodiment in particular, by correlating and evaluating the time-series information S indicating the user's situation at a specific timing with the model output data D corresponding to the element data presented to the user at the timing corresponding to that timing, it is possible to acquire analysis results that are easier for humans to understand, such as the influence that the user is receiving from the target content C.

[0036] FIG. 4 is a diagram conceptually illustrating an example of the analysis process performed by the analysis unit 23. The horizontal axis of the diagram represents the time axis, indicating that target content C was presented to user U during the time period from time t0 to time tN. In this example, time-series operation information indicating the details of operation inputs performed by user U is acquired for the period from time t0 to time tN. In this example, user U performs a predetermined operation (pressing an action button) at time tx, and operation information Sx indicating the details of that operation is recorded. In this case, the analysis unit 23 analyzes the details of content Cx presented to the user during a predetermined period T1 up to time tx, using multiple model output data Dx obtained from multiple frame images that constitute content Cx displayed within this period T1. The details may be the cause of the operation performed by user U at time tx. Therefore, by combining and using for analysis the time-series operation information at a predetermined timing with the model output data D obtained from the frame image data constituting the target content C presented to the user U at a timing corresponding to that timing (here, the period T1 preceding the time tx), it is possible to estimate the cause of the operation performed by the user U. Conversely, by performing an analysis using the model output data D targeting elements presented to the user U in a period after the predetermined time tx, the analysis unit 23 can estimate the impact that the operation performed by the user U had on the target content C.

[0037] Below, several specific examples of the analysis process performed by the analysis unit 23 will be described.

[0038] First, as a first example, an example of identifying the cause of a game interruption will be described. In this example, the target content C is a game play video, the element data is data of frame images constituting the game play video, and the game time-series information S is time-series operation information indicating the operations performed by the user U during game play.

[0039] In this example, when user U performs an operation to pause a game while playing the game, the analysis unit 23 identifies the timing of such a pause operation by referring to the time-series operation information. Then, the analysis unit 23 infers the reason why user U paused the game by using the contents of model output data D acquired by the model output data acquisition unit 22 based on image data of target content C output within a predetermined period preceding that timing. Specifically, if it can be read from the model output data D that an event that would be negatively evaluated by user U occurred immediately before the game was paused, there is a possibility that that event was the cause of the game pause.

[0040] More specifically, if the model output data D is information indicating vectors in the abstract space P as described above, the analysis unit 23 may analyze the cause of the user U's interruption of the game using the cluster distribution in the abstract space P. For example, the analysis unit 23 compares the distribution in the abstract space P of vectors obtained based on content presented at a predetermined time immediately before the user U interrupted the game with the distribution in the abstract space P of vectors obtained from content presented at another time period (i.e., a time period assumed to be a normal state in which no interruption cause has occurred). If a cluster is found to contain more vectors than in the normal state, the descriptive information corresponding to that cluster may represent the cause of the interruption. When extracting clusters representing such interruption causes, an evaluation process based on a natural language model or the like may be used to estimate whether each of the multiple clusters to be verified is likely to indicate a negative event that could cause the interruption, and clusters that may represent negative events may be extracted as targets for evaluation. If the number of vectors belonging to a cluster extracted in this manner is increased compared to the normal state, the descriptive information corresponding to that cluster is presumed to be the cause of the interruption.

[0041] As a second example, a case will be described in which the cause is identified when the user U repeatedly performs the same operation. In this second example, as in the first example, gameplay video and time-series operation information obtained while the user U is playing the game are analyzed.

[0042] If the user U repeatedly performs the same operation in a short period of time while playing a game, the operation may be a normal operation necessary for the game to proceed, but it may also be an operation performed because a problem has occurred in the game's progress, such as the game being unable to proceed. Therefore, the analysis unit 23 estimates the cause of such a repetitive operation by referring to the model output data D corresponding to the repetitive operation.

[0043] Specifically, the analysis unit 23 references the time-series operation information to detect an operation history in which the same operation is repeated a predetermined number of times or more within a predetermined time period. If such an operation history is detected, the analysis unit 23 acquires model output data D corresponding to the image data of the target content C presented to the user U during the period in which the operation history was detected and / or a predetermined period before and after that period. By analyzing this model output data D, the analysis unit 23 can infer the cause of the repeated operation. For example, if the user U repeatedly presses an operation button more than necessary, and if it is determined that the user U is in an unfavorable situation during a battle with an enemy in the scenes before and after that, it can infer that the cause of the operation is an attempt to overcome the unfavorable situation. Such an inference can be efficiently achieved not only by simply identifying characters and the like appearing in the image data, but also by inferring the meaning of the image from the model output data acquisition unit 22 and combining it with the time-series operation information.

[0044] The analysis unit 23 is not limited to the first and second examples described so far, but can extract operations that satisfy specific conditions from the time-series operation information, and by using the model output data D corresponding to the element presented to the user U at a timing determined according to the timing when the operation was performed, it can identify the content of the target content C that caused the operation, the content of the target content C caused by the operation, etc.

[0045] As a third example, an example in which the emotion of the user U is estimated based on the time-series information S and the relationship between the emotion of the user U and the target content C is analyzed will be described.

[0046] There are methods for estimating the emotions of a user U at any given time by using various types of time-series information S. For example, by capturing the facial expression of a user U while watching target content C using a camera and acquiring the resulting video as time-series information S, it is possible to estimate the emotions of the user U from changes in the facial expression. Furthermore, by acquiring data obtained by measuring the vital signs of the user U, such as heart rate, in real time as time-series information S, it is possible to identify whether the user U is nervous or excited. Furthermore, time-series operation information can also be used to estimate the emotions of a user. For example, if an operation such as repeatedly pressing a button is performed in a short period of time, it can be estimated that the user U may be excited.

[0047] When it is estimated that the user U is feeling a particular emotion based on the time-series information S, the analysis unit 23 acquires model output data D representing the content of the target content C presented to the user U during a period determined according to that timing (the period during which the user U is feeling the particular emotion and / or a predetermined period before and after that). By analyzing this model output data D, the analysis unit 23 can identify, as information meaningful to humans, the event that caused the user U to feel the particular emotion, the behavior that the user U will take when they feel the particular emotion, and so on.

[0048] Furthermore, the information about the user U's emotions obtained in this manner can also be used to estimate the cause of the interruption in the first example described above. For example, the analysis unit 23 identifies the time period before the user U interrupted playing the game when the user U felt negative emotions by identifying changes in the user U's emotions over time. Then, the analysis unit 23 obtains model output data D that represents the content of the target content C that was presented to the user U at that time. This allows the analysis unit 23 to analyze the cause of the user U interrupting playing the game at a finer level of granularity.

[0049] In the above description, the analysis unit 23 identifies time-series changes in the user U's emotions and associates the emotions with the model output data D based on the content of the target content C. However, the analysis unit 23 may identify various situations of the user U while viewing the target content C and perform the analysis process using the situation and the content of the target content C as evaluation targets. For example, by acquiring the output results of a sensor built into the operation device held by the user U while viewing the target content C as the time-series information S, the analysis unit 23 can identify the timing when the user U threw the operation device or put down the operation device. Furthermore, by acquiring video data from a camera capturing the state of the user U as the time-series information S, the analysis unit 23 can identify the timing when the user U was looking away or when the user U was away from their seat (i.e., when the target content C was being played but the user U was not viewing the target content C). The analysis unit 23 can estimate the cause of the user U's behavior by referring to the model output data D representing the content of the target content C at the timing corresponding to the attention timing identified based on the time-series information S.

[0050] As a fourth example, an example of specifying the content of the target content C that the user U focuses on will be described. In this example, the target content acquisition unit 21 acquires, as the time-series information S, information on the attention pattern of the user U with respect to the target content C. Here, the information on the attention pattern may include information on the attention position (information indicating which position the user U focuses on within the screen on which the target content C is presented), information on the attention level (information indicating how much the user U focuses on the target content C, i.e., how concentratedly the user U is watching the target content C), etc.

[0051] Information regarding the attention pattern can be acquired by capturing an image of the face of the user U and identifying the facial orientation and line of sight of the user U. Specifically, a technique for identifying the line of sight direction based on the position of the user U's pupils is known. Furthermore, it is possible to estimate the degree to which the user U is focused on the target content C based on information such as line of sight movement.

[0052] By referring to the information on the attention behavior obtained in this manner and the model output data D, the analysis unit 23 can identify the content of the target content C that is presented at a timing when it is estimated that the level of attention of the user U is high (or low). This makes it possible to analyze what kind of content attracts the interest of the user U, what kind of content is presented at what timing the user U becomes bored, and so on.

[0053] Furthermore, if the position of user U's attention can be identified, the content displayed in the area where user U is paying attention may be identified. As a specific example, the analysis unit 23 extracts a partial image from the frame image data constituting the target content C, the partial image being centered on the position where user U is particularly paying attention. Then, the model output data acquisition unit 22 uses the partial image as input and outputs the model output data D, which is acquired as information indicating the content where user U is paying attention. By performing such processing, it is possible to identify the object that is particularly attracting user U's interest within the frame image constituting the target content C.

[0054] As a fifth example, an example of generating summary information of target content C will be described. By using the model output data D, the analysis unit 23 can generate a summary of the target content C. Specifically, for example, the analysis unit 23 can identify the general flow of the content of the target content C along a time series by extracting model output data D corresponding to image data presented to the user at regular intervals. More specifically, the analysis unit 23 may generate a summary that explains the entire content by inputting multiple pieces of model output data D obtained at regular intervals into another machine learning model (for example, a model that generates natural language such as a large-scale language model).

[0055] The analysis unit 23 may also search for image data matching a predetermined linguistic expression to determine whether a certain event occurred or whether a certain object appeared in the target content C, and record the determined information as summary information. In this case, the recorded summary information may be linguistic description information or embedded data. Such summary information can be used to search for content containing a specific scene from among a large number of target contents C. In particular, by recording summary information using the model output data D, a user can search for the scene they want to see using natural language expressions, such as "Find the scene where someone is attacked by a zombie and breaks a window to escape."

[0056] When generating such summary information, the analysis unit 23 can use the time-series information S. For example, the analysis unit 23 uses the time-series information S to identify timings at which the user frequently performs input operations or timings at which the user's emotions change. Then, the analysis unit 23 obtains model output data D representing the content of the target content C corresponding to those timings, and generates summary information using the obtained model output data D. In this way, summary information including content that is considered to be particularly important to the user U can be generated.

[0057] As a sixth example, an example of generating statistical information of target content C will be described. By using model output data D obtained from element data constituting target content C, the analysis unit 23 can generate statistical information that tallies the number of times a specific event has occurred in target content C, etc.

[0058] Such aggregation can be achieved to some extent by simply referring to the model output data D acquired over the entire playback period of the target content C. However, by combining and utilizing the time-series information S, the analysis unit 23 can improve the accuracy of the obtained results and make the processing more efficient.

[0059] For example, when counting the number of times a user character operated by a user U in a game performs a specific action, the analysis unit 23 acquires corresponding model output data D from image data displayed at the timing corresponding to the user's operation to perform that action. This allows, for example, counting the number of times a user character shoots or passes in a soccer game. In particular, the analysis unit 23 can efficiently count the number of times an action is performed by referencing time-series operation information and determining, using the model output data D, whether the action was actually performed at the timing when the user U performed the operation to perform the action to be counted. Furthermore, by extracting model output data D indicating that the action was performed on the condition that time-series operation information indicating that the user U performed a predetermined operation is present, the analysis unit 23 can distinguish between actions by the user character and actions by other characters and count only actions resulting from the user character's operation.

[0060] The summary information and statistical information described above can also be used to add various types of metadata to the target content C or to perform more general-purpose data analysis. As a specific example, by referencing statistical information obtained from multiple target contents C, it is possible to tally up the overall number of times a specific action was performed or to determine the percentage of play data that triggers a specific event.

[0061] Furthermore, the analysis unit 23 can use a method similar to that for acquiring summary information and statistical information as described above to perform a qualitative evaluation of the content of the user U's gameplay, rather than being limited to conventional quantitative evaluations. For example, by extracting model output data D that indicates whether the player attacked the enemy from a blind spot or used a decoy to outwit the enemy, the analysis unit 23 can evaluate the skill and proficiency of the user U's play, which cannot be evaluated simply by statistical information such as the number of wins or hits. Such evaluation results can be used for rating the user U and for matching users together.

[0062] This information can also be used to analyze the play style of user U. For example, in a cooperative game, if the user character controlled by user U acts at a distance from the companion characters controlled by other users, conventionally, user U might be estimated to be an uncooperative player simply based on information about the average distance between the characters. However, if the summary information can identify that the user character usually keeps a distance from the companion characters but takes action to help them when the companion characters are in trouble, user U can be estimated to be a cooperative player. In addition, by obtaining model output data D representing the user character's behavior, it is possible to evaluate whether the play style is aggressive (or passive), whether the play style is one that progresses through the game quickly to achieve results, or whether the play style is one that progresses slowly.

[0063] In the explanation so far, the main example has been target content C whose content changes in response to operation inputs by the user U, such as gameplay video, and therefore it has been assumed that one user U views one piece of target content C, and that the target content acquisition unit 21 acquires time-series information S related to that user U. However, the target content C that the information processing device 10 according to the present embodiment analyzes is not limited to content viewed by only one such user U.

[0064] For example, the target content C may be a video stream distributed via a communication network such as the Internet, or a game in which multiple people participate via a communication network. In this case, multiple users U will view the target content C simultaneously or at different times. In such an example, the target content acquisition unit 21 may acquire time-series information S representing the operation details, vital information, etc., for each of the multiple users U, and the analysis unit 23 may analyze the target content C based on the acquired multiple pieces of time-series information S.

[0065] Furthermore, the time-series information S may be statistical information regarding a plurality of users U who simultaneously view the target content C. Hereinafter, such an example will be described as a seventh example.

[0066] In this example, the target content C is content that is simultaneously distributed over a communication network, such as live video. The target content acquisition unit 21 acquires viewer count information, as time-series information S, that indicates how the number of viewers of the target content C has changed in real time when the target content C is distributed.

[0067] The analysis unit 23 refers to the viewer count information as the time-series information S to identify parts of the target content C that were particularly viewed (periods with a large number of viewers) and, conversely, parts that were not viewed (periods with a small number of viewers). The analysis unit 23 may also identify timings of significant changes in the number of viewers, such as timings when the number of viewers increased or decreased. By acquiring model output data D indicating the content presented to viewers at timings corresponding to the identified timings, the analysis unit 23 can estimate the causes of what content attracts attention, or what content causes viewers to stop watching and leave, etc.

[0068] In the above description, the target content C is assumed to be content actually viewed or played by the user U. However, the target content C may also be content generated by an automatic execution program or the like. For example, for the purpose of verifying the operation of a target program such as a game program, the automatic execution program may automatically execute the target program by simulating the same operational inputs as a human. After such automatic execution, it is necessary to confirm whether the target program operated normally. However, visual confirmation by a human is extremely time-consuming. Furthermore, when attempting to automatically determine whether the result of automatic execution is successful or not by a computer, different measures are required for each target program, such as precisely defining the criteria for determination, making efficient execution difficult. Therefore, the information processing device 10 according to this embodiment may use the video output as a result of automatic execution by the automatic execution program as the target content C, and perform an analysis process on the video to estimate whether the target program operated normally. Such an example is described below as an eighth example.

[0069] For example, the analysis unit 23 generates summary information and statistical information for the target content C resulting from the automatic execution, as in the fifth and sixth examples described above, and verifies whether the results are in line with the intended content of the automatic execution. Even if the verification itself is performed by a human, checking whether the intended play result has been output by, for example, referring to the summary information is easier than visually checking the target content C itself. Furthermore, if keywords related to the play result are defined in advance, the analysis unit 23 can verify whether the result has been obtained by referring to the summary information. As a specific example, in a quest-type game, when gameplay is automatically performed to achieve an objective, such as obtaining a specific item or defeating a specific enemy, if descriptive information indicating the content corresponding to the objective is included in the model output data D obtained from the target content C, it can be determined that the play to achieve the objective is being performed correctly. Furthermore, if the accuracy of the semantic analysis contained in the content can be improved, the analysis unit 23 itself can perform analysis based on images of scenes in which a character is given a task in the game to identify the objective that the player must achieve, and then evaluate the success or failure of automatic execution by analyzing whether the player is performing actions that are appropriate to the identified objective based on images of subsequent scenes and chronological operation information that indicates the content of the operation input entered by the automatic execution program.

[0070] As described above, according to the information processing device 10 of this embodiment, by utilizing the model output data D and time-series information S obtained from the element data that constitutes the target content C, it is possible to effectively realize analysis that is more meaningful to humans.

[0071] Note that the embodiments of the present invention are not limited to those described above. For example, in the above description, frame image data constituting a moving image is primarily used as element data constituting the target content C. However, the present invention is not limited to this. The machine learning model M may also include a machine learning model obtained by learning the relationship between audio data and descriptive information that linguistically describes the content of that audio, or a machine learning model that accepts time-series information itself as a learning subject and learns the relationship between the time-series information and image data, etc. In any case, by using the model output data D output by the machine learning model M obtained by learning the relationship between the elements constituting content that changes over time and the descriptive information that linguistically describes that content, the information processing device 10 according to this embodiment can realize an analysis process that enables the content of the target content C to be understood in a more meaningful way for humans.

[0072] In the above description, the information processing device 10 according to the embodiment of the present invention stores the machine learning model M itself, and acquires the model output data D by inputting element data into the machine learning model M. However, this is merely an example. The information processing device 10 may acquire the model output data D by using a machine learning model M held by another computer connected via a communication network, for example.

[0073] REFERENCE SIGNS LIST 10 information processing device, 11 control unit, 12 storage unit, 13 interface unit, 14 display device, 15 operation device, 21 target content acquisition unit, 22 model output data acquisition unit, 23 analysis unit

Claims

1. An information processing device having one or more processors, wherein the one or more processors: acquire target content having content that changes over time and specified time series information recorded while the target content is being output; acquire model output data corresponding to the element data that constitutes the target content using a machine learning model obtained by machine learning the relationship between element data included in the content and descriptive information that linguistically describes the content of the element data; and execute specified processing based on the acquired time series information and the acquired model output data.

2. An information processing device according to claim 1, wherein the target content is content presented to a user, and the time series information is information indicating a state of the user while viewing the target content.

3. An information processing device according to claim 2, wherein the one or more processors acquire, as the time series information, information including at least one of the facial expression of the user, the behavior of the user, and vital information of the user while viewing the target content.

4. An information processing device according to claim 2, wherein the target content includes moving images that change depending on the content of the operation input performed by the user, and the one or more processors acquire, as the time series information, information indicating the content of the operation input performed by the user while viewing the target content, identify the time point at which a specified operation was performed by the user based on the information indicating the content of the operation input, acquire the model output data corresponding to the element data presented to the user at a timing selected depending on the time point at which the specified operation was performed, and execute the specified processing relating to the relationship between the specified operation and the target content based on the acquired model output data.

5. An information processing device according to claim 4, wherein the specified operation is an interruption operation that interrupts the output of the target content, and the one or more processors execute, as the specified processing, a process of estimating the cause of the interruption operation based on the model output data corresponding to the element data presented to the user during a period preceding the interruption operation.

6. An information processing device according to claim 4, wherein the specified operation is a repetitive operation in which the same operation is repeated a specified number of times or more, and the one or more processors execute, as the specified processing, a process of estimating a cause of the repetitive operation based on the model output data corresponding to the element data presented to the user in a period preceding the repetitive operation.

7. An information processing device according to claim 2, wherein the one or more processors infer time-series changes in the emotions of the user while watching the target content based on the user's state indicated by the time-series information, and execute the specified processing based on the inferred results.

8. An information processing device according to claim 2, wherein the one or more processors acquire information relating to the user's attention to the target content as the time series information, and perform, as the predetermined processing, a process of identifying the content of the target content that the user is paying attention to based on the user's attention to the target content.

9. An information processing device according to claim 1, wherein the target content is content presented to a plurality of viewers, and the one or more processors acquire, as the time series information, information indicating changes over time in the number of viewers of the target content, identify a time point to be verified based on the information indicating changes over time in the number of viewers, acquire the model output data corresponding to the element data presented to the plurality of viewers at a timing selected according to the identified time point, and execute, as the specified processing, a process of estimating the status of the viewers at the time point to be verified based on the acquired model output data.

10. An information processing device according to claim 1, wherein the one or more processors execute, as the predetermined processing, a process of generating statistical information that counts the number of times an event to be aggregated has occurred based on the acquired model output data.

11. An information processing device according to claim 10, wherein the one or more processors count the number of times the event to be counted occurs at a timing specified based on the time-series information.

12. An information processing device according to claim 1, wherein the target content is an image obtained by automatically executing a program to be verified using an automatic execution program, and the one or more processors, as the specified processing, execute a process of evaluating a result of the automatic execution of the automatic execution program based on the acquired model output data.

13. An information processing method comprising: acquiring target content having content that changes over time and specified time series information recorded while the target content is being output; acquiring model output data corresponding to the element data that constitutes the target content using a machine learning model obtained by machine learning the relationship between element data included in the content and descriptive information that linguistically describes the content of the element data; and executing specified processing based on the acquired time series information and the acquired model output data.

14. A program for causing a computer to execute a process which includes acquiring target content having content that changes over time and specified time series information recorded while the target content is being output, acquiring model output data corresponding to the element data constituting the target content using a machine learning model obtained by machine learning the relationship between element data contained in the content and descriptive information that linguistically describes the content of the element data, and executing specified processing based on the acquired time series information and the acquired model output data.

Images