Information processing device, information processing method, and program

Abstract

This information processing device: sequentially acquires screen data related to details of a screen of a target content having details that change with time, while the target content is being outputted; sequentially acquires model output data corresponding to the screen data related to details of the screen of the target content by using a machine learning model obtained by learning the relationship between the screen data and description information describing details of the screen data in a linguistic manner; and executes prescribed processing on the basis of the acquired model output data.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program for generating moving images.

[0002] When a user plays a video game, for example, the information processing device sequentially generates content (moving images and audio) that changes over time in response to the user's operational inputs and behavior, and presents it to the user.

[0003] When presenting the above-described content, it is desirable for the information processing device to present content that is appropriate for the situation or circumstances, such as by advising the user on what operations to perform or by reacting to the user's behavior. However, simply presenting content determined based on pre-defined conditions can result in the presented content being monotonous, or it can require setting complex conditions in advance to present a variety of content. This makes it difficult to present content that is appropriate for each individual user's situation.

[0004] The present invention has been made in consideration of the above-mentioned circumstances, and one of its objects is to provide an information processing device, an information processing method, and a program that can present guidance, responses, etc. according to the user's situation.

[0005] An information processing device according to one aspect of the present invention is an information processing device that includes one or more processors, and the one or more processors sequentially acquire screen data related to the screen content of target content having content that changes over time while the target content is being output, sequentially acquire model output data corresponding to the screen data related to the screen content of the target content using a machine learning model obtained by machine learning the relationship between the screen data and descriptive information that linguistically describes the content of the screen data, and execute a predetermined process based on the acquired model output data.

[0006] An information processing method according to one aspect of the present invention is an information processing method that, for target content having content that changes over time, sequentially acquires screen data related to the screen content of the target content while the target content is being output, sequentially acquires model output data corresponding to the screen data related to the screen content of the target content using a machine learning model obtained by machine learning the relationship between the screen data and descriptive information that linguistically describes the content of the screen data, and performs predetermined processing based on the acquired model output data.

[0007] A program according to one aspect of the present invention causes a computer to execute a process, which includes: sequentially acquiring screen data relating to the screen content of target content having content that changes over time while the target content is being output; sequentially acquiring model output data corresponding to the screen data relating to the screen content of the target content using a machine learning model obtained by machine learning the relationship between the screen data and descriptive information that linguistically describes the content of the screen data; and executing predetermined processing based on the acquired model output data. This program may be provided by being stored on a computer-readable, non-transitory information storage medium.

[0008] It is a configuration block diagram showing the configuration of an information processing device according to an embodiment of the present invention. It is a functional block diagram showing the functions of an information processing device according to an embodiment of the present invention. It is a diagram for explaining an example of a learning process for generating a machine learning model used by the information processing device according to an embodiment of the present invention. It is a diagram for explaining an example of a process in which the information processing device according to an embodiment of the present invention presents guidance information.

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0010] 1 is a block diagram showing the configuration of an information processing device 10 according to one embodiment of the present invention. The information processing device 10 is a personal computer, a server computer, or the like, and as shown in the figure, includes a control unit 11, a storage unit 12, and an interface unit 13. The information processing device 10 is also connected to a display device 14 and an operation device 15.

[0011] The control unit 11 includes at least one processor such as a CPU, and performs various information processing by executing programs stored in the storage unit 12. Specific examples of the processing performed by the control unit 11 in this embodiment will be described later. The storage unit 12 includes at least one memory device such as a RAM, and stores the programs executed by the control unit 11 and data processed by the programs.

[0012] The interface unit 13 is an interface for data communication between the display device 14 and the operation device 15. The information processing device 10 is connected to the display device 14 and the operation device 15 via the interface unit 13 either wired or wirelessly. Specifically, the interface unit 13 includes a multimedia interface for transmitting a video signal supplied by the information processing device 10 to the display device 14. It also includes a data communication interface for receiving a signal indicating the operation content performed by the user on the operation device 15.

[0013] The display device 14 displays on its screen an image corresponding to a video signal supplied from the information processing device 10. The operation device 15 is, for example, a keyboard or a mouse, and receives operation input from a user. The operation device 15 is connected to the information processing device 10 by wire or wirelessly, and transmits an operation signal indicating the content of the operation input received from the user to the information processing device 10.

[0014] 1, the information processing device 10 may be connected to various devices that acquire information from the user in real time, such as a microphone or a camera, and may be configured to be capable of communicating with information processing devices such as terminal devices and server devices used by other users.

[0015] The functions realized by the information processing device 10 will be described below with reference to the functional block diagram of FIG. 2. As shown in FIG. 2, the information processing device 10 is functionally configured to include a content output unit 21, a screen data acquisition unit 22, a model output data acquisition unit 23, and a guidance presentation unit 24. These functions are realized by the control unit 11 operating in accordance with one or more programs stored in the storage unit 12. These programs may be provided to the information processing device 10 via a communication network such as the Internet, or may be provided by being stored on a computer-readable information storage medium such as an optical disk.

[0016] The content output unit 21 is realized by executing an application program, and generates and outputs content to be presented to the user. For example, the content output unit 21 is realized by a game program. In this case, the content output unit 21 executes game processing in response to user input operations, and renders and outputs a game screen showing the results. Hereinafter, the content output by the content output unit 21 will be referred to as target content C. In this embodiment, this target content C is content having content that changes over time, and includes at least moving images presented to the user. In addition, hereinafter, a user who uses the information processing device 10 to view target content C in real time will be referred to as user U.

[0017] The target content C presented to the user U by the content output unit 21 may include a moving image depicting a virtual three-dimensional space. In this case, the content output unit 21 determines the appearance of various objects to be placed in the virtual three-dimensional space and instructs the drawing engine to draw a spatial image showing the virtual three-dimensional space. The spatial image drawn in response to this drawing instruction is displayed on the display device 14 as a frame image constituting the target content C.

[0018] The program that realizes the content output unit 21 may be a program separate from or the same as the programs that realize the screen data acquisition unit 22, the model output data acquisition unit 23, and the guidance presentation unit 24. As a specific example, the following describes a case where the screen data acquisition unit 22, the model output data acquisition unit 23, and the guidance presentation unit 24 are realized by a program independent of the content output unit 21.

[0019] The screen data acquisition unit 22 sequentially acquires screen data relating to the contents of the screen of the target content C output by the content output unit 21. As a specific example, the screen data may be the frame images of a moving image drawn by the content output unit 21 and displayed on the display device 14. Furthermore, the screen data acquired by the screen data acquisition unit 22 may be drawing data output by the content output unit 21 to draw the state of the virtual three-dimensional space. In this case, the screen data includes information specifying the appearance, position, orientation, etc. of various objects placed in the virtual space.

[0020] For example, at predetermined time intervals, the screen data acquisition unit 22 acquires the latest screen data at that time from the content output unit 21. In this way, the screen data acquisition unit 22 acquires in real time screen data relating to the content of the screen presented to the user U in chronological order.

[0021] Furthermore, the screen data acquisition unit 22 may sequentially acquire time-series information indicating the status of the user U from the content output unit 21. As a specific example, the time-series information S may be time-series operation information indicating the content of operation inputs made to the operation device 15 by the user U watching target content C such as a game while the target content C is being output. The time-series operation information includes information that can identify the content of the operation made by the user U (for example, a pressed button, or the direction and amount of operation of a stick-type operation member) and the timing of the operation.

[0022] Furthermore, the time-series information is not limited to time-series operation information, but may be various types of information related to the status of the user U while the target content C is being output. For example, the time-series information may include information obtained by recording in real time, using various sensors, vital signs (e.g., heart rate) and behavior (e.g., body movements) of the user U while watching the target content C. The time-series information may also include information such as the user U's facial expressions and eye movements obtained by capturing an image of the user U with a camera. It may also include audio data obtained by recording what the user U has said with a microphone. The screen data acquisition unit 22 may also acquire information obtained by analyzing such real-time recorded information as time-series information. For example, by analyzing an image captured of the user U or information related to the user U's body movements, the target content acquisition unit 21 may acquire, as time-series information, information indicating that the user U left their seat at a predetermined time, information indicating the user U's emotions at each point in time, information regarding the degree to which the user U was concentrating on the target content C, information indicating which part of the target content C the user U was paying attention to, and the like.

[0023] By referencing this target content C in combination with time series information, the information processing device 10 can identify the content of the operation performed by user U at the time a certain image is displayed, as well as the situation of user U at that time, such as user U's facial expression, posture, and emotions.

[0024] The model output data acquisition unit 23 uses a pre-prepared machine learning model M to acquire model output data D relating to the content of the screen data sequentially acquired by the screen data acquisition unit 22. This machine learning model M is a machine learning model obtained by learning the relationship between the screen data and descriptive information used to linguistically describe the content.

[0025] The machine learning model M used in this embodiment may be a model based on various techniques as long as it can acquire information used to linguistically describe the content of the elements that make up the target content C. For example, if the analysis target is image data showing a horse galloping with a man on it, it is desirable that this machine learning model M not simply output the names and positions of objects (horse, man, etc.) contained in the image, but rather a model that ultimately obtains a descriptive sentence such as "A samurai is galloping on horseback through the wilderness." A machine learning model M capable of such output can be generated, for example, by a technique such as CLIP (Contrastive Language-Image Pre-training).

[0026] However, in this embodiment, the model output data D acquired by the model output data acquisition unit 23 as output from the machine learning model M does not need to be descriptive sentences that humans can understand, but may be information corresponding to linguistic information. An example of a learning process for generating the machine learning model M used by the information processing device 10 according to this embodiment will be described below with reference to FIG. 3. Note that, although a case where the screen data is two-dimensional image data will be described here, a similar learning process may be performed to generate the machine learning model M even when the screen data is drawing data for drawing a spatial image showing the appearance of a virtual three-dimensional space.

[0027] The learning processing device that executes the learning process in this example accepts multiple sets of training data, each set consisting of image data and description information that linguistically describes the image data. For each set of training data, the learning device divides the description information contained in a given set into tokens (e.g., words) t and obtains its distributed representation tref (token embedding). The learning processing device 10 then generates description-related data ft(tref) by encoding the tokens using a text encoder. This description-related data ft(tref) is multidimensional vector data and corresponds to one piece of coordinate information in an abstract space.

[0028] The learning processing device also generates image-related data fi(I) by inputting image data I included in the same set as the description information T into an image encoder. The learning processing device sets the image encoder so that this image-related data fi(I) also becomes vector data of the same dimension as the description-related data ft(tref). This image-related data fi(I) also corresponds to one piece of coordinate information in the abstract space.

[0029] The learning processing device performs machine learning on each encoder ft, fi so that the description-related data ft(tref) and the image-related data fi(I) are arranged in close proximity to each other in the predetermined abstract space (a vector-dimensional space of the description-related data and the image-related data, known as a multimodal embedding space in CLIP). Conceptually, this results in the image data and the description information describing it being associated with each other in the predetermined abstract space P (S11), as shown in the example of Figure 3. In the above example, the text encoder ft may use a well-known transformer network, and the image encoder fi may use a Vision Transformer (ViT).

[0030] In this embodiment, data of the machine learning model M generated by such machine learning is stored in advance in the storage unit 12. The model output data acquisition unit 23 inputs data of frame images constituting the target content C acquired by the screen data acquisition unit 22 to the machine learning model M, and outputs the output as model output data D.

[0031] When using the machine learning model M that converts each element into a vector in the abstract space P described above, the model output data D may be the image-related data fi(I) itself obtained by inputting frame image data I that constitutes the target content C into an image encoder fi. Alternatively, the model output data D may be data relating to a character string representation obtained by inputting this image-related data fi(I) into a decoder that performs the reverse conversion to a text encoder that converts tokens corresponding to descriptive information T into a vector representation. Such model output data D becomes data that more directly represents words and sentence representations related to the frame image data I.

[0032] Alternatively, an index value (such as cosine similarity) indicating the distance in the abstract space P between the description-related data ft(tref) and the image-related data fi(I) corresponding to previously prepared description information T may be calculated, and the evaluation result using this index value may be used as the model output data D. By using such an index value, it is possible to evaluate the extent to which the frame image data I corresponds to the content expressed by the predetermined description information T. For example, by preparing multiple pieces of description information T related to violent expressions in advance and evaluating the degree to which the frame image data I to be analyzed is close to these pieces of description information T in the abstract space P, it is possible to determine whether the frame image data I contains violent expressions.

[0033] Furthermore, the model output data D may be data obtained by performing clustering on the distribution within an abstract space P obtained by converting multiple image data into vector representations. For example, a clustering process is performed in which multiple image-related data fi(I) obtained by inputting multiple image data extracted under predetermined conditions into an image encoder fi is grouped within the abstract space P. If, for example, three clusters are extracted as a result, it can be estimated that the image data belonging to each of these clusters are images with similar meanings. Furthermore, description information T corresponding to the position of each cluster may be obtained as the model output data D. This description information T can be used to roughly explain the content of the image data belonging to the corresponding cluster.

[0034] In any case, the model output data D obtained by inputting certain image data into the machine learning model M is data that can be used to represent the content of the image data in a manner that can be understood by humans. The guidance presentation unit 24, which will be described later, uses this model output data D to present guidance regarding the content of the target content C.

[0035] Note that the machine learning model M described above is merely an example, and in this embodiment, the model output data acquisition unit 23 can use various models that can obtain the meaning of an image or descriptive information about the image as the machine learning model M. For example, the model output data acquisition unit 23 may use, as the machine learning model M, a model that has been trained to obtain answers to questions about an image by learning images, questions about the images, and the answers to those questions. Furthermore, the machine learning model M may use, as the machine learning model M, a model that has been trained to extract information about a specific perspective by inputting sentences or the like that condition the perspective for information extraction along with the image. By using such a machine learning model, the guidance presentation unit 24, which will be described later, can present guidance information that focuses on a specific perspective.

[0036] The guidance presentation unit 24 executes a guidance process in which it determines what information to provide to the user U based on the model output data D acquired by the model output data acquisition unit 23, and presents the determined information as guidance information to the user U. By determining the guidance information using the model output data D, the guidance presentation unit 24 can be expected to provide guidance that is appropriate for the content being presented to the user U at that time and the situation of the user U, even if detailed rules for determining the guidance information are not defined in advance.

[0037] Specifically, for example, the guidance presentation unit 24 presents a message to the user U, such as a character string of advice for the user U or a supplementary explanation of the current content. This message may be displayed as a message image superimposed on the screen displayed by the content output unit 21, or may be played as a message voice. Furthermore, the guidance presentation unit 24 may present guidance to the user U by displaying on the screen, such as a display indicating a specific position on the screen displayed by the content output unit 21.

[0038] Below, several specific examples of the process in which the guidance presentation unit 24 presents guidance to the user U will be described.

[0039] As a first example, an example will be described in which a conversation regarding the contents of the screen is realized with the user U. In this example, the guidance presentation unit 24 functions as a virtual agent capable of communicating with the user U.

[0040] In this example, the guidance presentation unit 24 determines the content of the conversation to be presented to the user U at a predetermined timing, such as when a certain amount of time has passed, when the scene presented on the screen changes, or when the voice spoken by the user U is recognized, and displays the determined content of the conversation on the screen as a message image or plays it back as a message voice.

[0041] Specifically, the guidance presentation unit 24 determines the content of the conversation to be presented using the model output data D acquired based on screen data presented to the user U within a predetermined period of time in the most recent past. By determining the content of the conversation using the model output data D in this manner, the guidance presentation unit 24 can present to the user U conversation related to the content most recently displayed on the screen, the most recent action of the character operated by the user U, and the like.

[0042] In this case, it is desirable that the guidance presentation unit 24 presents a message including impressions and emotional cognitive content about the content as a conversational sentence, rather than simply describing the content recognized from the screen data in an explanatory manner. For example, the guidance presentation unit 24 can obtain such a message by inputting the model output data D obtained based on the screen data into the second machine learning model M2. In this example, the second machine learning model M2 is a model trained by machine learning to generate a conversational sentence, and may be realized by a known method such as generation AI.

[0043] FIG. 4 conceptually illustrates the process of the guidance presentation unit 24 for generating such conversational sentences. In this example, model output data D obtained as the output of the machine learning model M is further input to a second machine learning model M2, thereby generating conversational sentences containing emotional cognitive content about the content of the screen data. Using this method, the guidance presentation unit 24 can present user U with conversational sentences that sound as if a human with emotions were watching the same content as user U, such as, "The item is under the tree," "Everyone looks like they're having fun drinking and singing," or "That was close. It was surprising when they suddenly attacked from the roof." This allows user U to enjoy a conversation with a virtual agent while watching the content.

[0044] Note that the guidance presentation unit 24 may determine guidance information to be presented to the user U by performing predetermined processing on the model output data D of the machine learning model M without using the second machine learning model M2. As a specific example, when a model that outputs descriptive information describing screen data from a specific perspective (e.g., answers to specific questions) is used as the machine learning model M as described above, a machine learning model M trained to generate answers to anticipated user questions can be prepared in advance. In this case, when an anticipated question is received from the user, the guidance presentation unit 24 may present the content of the model output data D obtained using this machine learning model M itself, or a message including that content, as guidance information to the user.

[0045] Furthermore, the machine learning model M may be configured to output the model output data D in the form of sentences spoken to the user, according to pre-specified content, thereby enabling the user to obtain guidance information that sounds as if it were being spoken by an agent having a conversation with the user.

[0046] As a second example, an example will be described in which advice regarding the content of a game is presented as guidance information while the user U is playing the game.

[0047] In this example, the guidance presentation unit 24 first determines whether the user U is in a situation where he or she needs advice based on given conditions. Specifically, for example, the guidance presentation unit 24 estimates that the user U needs advice when the user U is repeatedly performing the same operation or when it is estimated that the game is not progressing and the same scene is continuing.

[0048] As a more specific example, the guidance presentation unit 24 may refer to time-series operation information indicating the content of operation inputs by the user U, and may present advice when it is determined that the user U is repeating the same operation. Furthermore, the guidance presentation unit 24 may refer to model output data D that is periodically acquired, and may present advice when it is determined from the content of the model output data D that the same scene is being repeated or that no change in scene has occurred for a predetermined time or more. Furthermore, when it is possible to estimate the emotions of the user U based on time-series information obtained by capturing images of the facial expressions of the user U with a camera, for example, advice may be presented when it is estimated that the user U is irritated.

[0049] When it is determined that advice should be provided, the guidance presentation unit 24 estimates the situation in which the user U is placed by referring to the model output data D and determines advice appropriate to that situation. For example, if there is a signboard at a fork in the road on the screen presented to the user U, it is possible to present advice such as "Check the signboard there." Furthermore, if the user U is taking damage despite defending against an enemy's fire attack, it is possible to present advice such as "You need to avoid the fire." In this second example, as in the first example, the content of the advice may be realized by a second machine learning model M2 that determines the content of the advice using the model output data D.

[0050] In this way, the guidance presentation unit 24 uses the model output data D acquired from the screen data being presented to the user U in real time to recognize the meaning of the screen being presented to the user U and determines advice content in accordance with that content. This makes it possible to provide the user U with natural advice and specific advice that is in line with the content of the screen, even if the user does not have a prior understanding of the game content.

[0051] As a third example, an example of supporting communication between multiple users will be described. In this example, multiple users communicate with each other while sharing a single piece of content, such as when participating in a multiplayer game. Here, "multiple users sharing a single piece of content" refers to a situation in which mutually related screens are presented to each user, such as when multiple users are viewing the same screen or when each user is operating a character located in the same virtual space.

[0052] In such cases, each user may communicate with other users by sending short predefined phrases, gestures (e.g., emotes), emoticons, stamps, etc. Hereinafter, the various actions that users send to other users for communication purposes are collectively referred to as "messages." Furthermore, among messages, a predefined message refers to a message that a user U can select and send from multiple candidates prepared in advance by a program that outputs the target content C. While such predefined messages have the advantage of being easy to input, they may be limited to predefined types and lack expressiveness. Therefore, when user U performs an operation to send a predefined message to another user, the guidance presentation unit 24 may generate a message with content appropriate to the situation based on the model output data D, and send the generated message as a supplement to the predefined message specified by user U, or as a substitute for the predefined message specified by user U.

[0053] As a specific example, when the user U instructs another user who has defeated an enemy that was attacking the user U to send a standard message such as "Thank you," the guidance presentation unit 24 may generate a message such as "Wow, you saved me," and send it to the user specified as the destination. Alternatively, when the character controlled by the user U sends the standard message "Yes," immediately after placing an item that the user U is holding on the ground, the guidance presentation unit 24 may generate and send a message such as "I'll give this to you."

[0054] To achieve this processing, the guidance presentation unit 24 may use another machine learning model M3 that receives as input the model output data D and a standard message selectable by the user U. In this example, the machine learning model M3 is a model trained by machine learning to receive as input the model output data D, which is the output of the machine learning model M, and the standard message, and to generate and output a message that can be used in the situation indicated by the model output data D as an alternative or supplement to the standard message.

[0055] Furthermore, in this example, multiple machine learning models M3 may be prepared. In this case, the guidance presentation unit 24 selects a machine learning model M3 to use depending on the user to whom the message is to be sent, and inputs the message specified by the user U and the model output data D into that model to generate a message to be sent to the user. In this case, the multiple machine learning models M3 may be models generated by learning messages exchanged in different countries or regions. Differences in culture and customs between countries or regions, such as the interpretation of a certain gesture as having a different meaning depending on the country, can lead to differences in the meaning of a message and the situations in which it is used. Therefore, multiple machine learning models M3 obtained by performing machine learning on responses between users in each region are prepared. When the user U instructs to send a message, the guidance presentation unit 24 references the attribute information of the user specified as the message recipient and selects a machine learning model M3 corresponding to the region to which the user belongs. The selected machine learning model M3 then inputs the specified template message and the model output data D into the selected machine learning model M3 to generate the message to be sent. In this way, the user U, who is sending the message, can absorb cultural differences and send a message appropriate to the recipient's country or region without being particularly aware of them. In this case, the multiple machine learning models M3 do not need to be generated independently from one another, but may be generated by performing fine tuning on a single machine learning model to additionally learn content related to each region.

[0056] As in the first example described above, the guidance presentation unit 24 may present guidance information using the model output data D of the machine learning model M directly, rather than using a separate machine learning model M3. In this case, a machine learning model M that has previously learned the association between screen data and a message corresponding to the situation of the scene represented by the screen data is used. Furthermore, instead of preparing multiple independent machine learning models for each country or region, a message may be generated using a single machine learning model M or machine learning model M3 that has previously been trained to enable output tailored to multiple countries or regions. In this case, the guidance presentation unit 24 provides attribute information of the destination user to the machine learning model to be used in advance, so that the output is tailored to the destination user of the target message, and then transmits a message output according to the attribute information to the destination user. In this way, a message tailored to the recipient's attributes can be presented without preparing multiple machine learning models.

[0057] Furthermore, the guidance display unit 24 may not only send supplemental or alternative messages in response to a message transmission instruction from user U, but may also display guidance to supplement the content of messages exchanged between users. For example, if user U sends another user a message containing a demonstrative term such as "Let's go to that mountain hut," the guidance display unit 24 uses the model output data D to identify the target indicated by the demonstrative term contained in the screen data. Then, the guidance display unit 24 transmits user U's message along with information identifying the object in the virtual space that is the target of the message (here, the mountain hut) to the device used by the other user. The user device that receives this message highlights the specified object (here, the mountain hut), for example, by displaying a pin on the specified object. This allows other users who receive a message from user U to easily understand the intent of the message and facilitate smooth communication.

[0058] Furthermore, instead of or in addition to highlighting an object, the guidance presentation unit 24 may generate a message explaining the position of the object and present it to other users as guidance information. This message may include text explaining the position coordinates, direction, distance, etc. of the object.

[0059] In this example, if the screen data acquired by the screen data acquisition unit 21 is not a frame image presented to the user U but drawing data containing information about an object in the virtual space, it is desirable that the guidance presentation unit 24 identify the object itself designated by the user U based on the correspondence between the model output data D and the drawing data. This makes it easy to transmit and share information about the identified object to terminals used by other users.

[0060] As described above, according to the information processing device 10 of this embodiment, by utilizing the model output data D obtained from the screen data presented to the user U, it is possible to present the user U with guidance and responses that are appropriate to the user U's situation at that time.

[0061] It should be noted that the embodiments of the present invention are not limited to those described above. For example, in the above description, the machine learning model M receives screen data as input and outputs model output data D. However, the machine learning model M may also receive, as input, not only screen data but also various other data related to the target content C, such as voice data and time-series information on the operation details received from the user.

[0062] Furthermore, the model output data acquisition unit 23 may acquire the model output data D by inputting a portion of the screen data extracted based on predetermined conditions into the machine learning model M, rather than inputting the entire screen data. Specifically, for example, the model output data acquisition unit 23 may extract an area determined according to the position of the user character from a frame image presented to the user, and input the extracted area to the machine learning model M. Furthermore, the model output data acquisition unit 23 may extract information about objects included in an area determined according to the position of the user character and / or other objects (e.g., characters controlled by other users playing the game together) in the virtual space, and input the extracted information to the machine learning model M. With this type of control, it is possible to acquire model output data D related to the content of an area, particularly an area related to a character controlled by the user or an area expected to attract the user's attention.

[0063] In the above description, the information processing device 10 according to the embodiment of the present invention stores the machine learning model M itself, and acquires the model output data D by inputting screen data into the machine learning model M. However, this is merely an example. The information processing device 10 may acquire the model output data D by using a machine learning model M held by another computer connected via a communication network, for example.

[0064] REFERENCE SIGNS LIST 10 information processing device, 11 control unit, 12 storage unit, 13 interface unit, 14 display device, 15 operation device, 21 target content output unit, 22 screen data acquisition unit, 23 model output data acquisition unit, 24 guidance presentation unit

Claims

1. An information processing device having one or more processors, wherein the one or more processors sequentially acquire screen data relating to the screen content of target content having content that changes over time while the target content is being output, sequentially acquire model output data corresponding to the screen data relating to the screen content of the target content using a machine learning model obtained by machine learning the relationship between the screen data and descriptive information that linguistically describes the content of the screen data, and execute a specified process based on the acquired model output data.

2. An information processing device according to claim 1, wherein the one or more processors further acquire predetermined time series information while the target content is being output, and generate guidance information for a user viewing the target content based on the acquired time series information and the acquired model output data, and present the information to the user.

3. An information processing device as described in claim 1, wherein the one or more processors input the acquired model output data into a second machine learning model that has been trained by machine learning to generate conversational sentences, and present the resulting conversational sentences to a user viewing the target content.

4. An information processing device as described in claim 1, wherein the one or more processors receive a transmission instruction from a user viewing the target content to send a predetermined standard message to another user, determine the content of the message to be sent to the other user based on the acquired model output data and the standard message specified in the transmission instruction, and send the determined message to the other user.

5. An information processing device according to claim 4, wherein the one or more processors input the acquired model output data and a standard message specified in the sending instruction to a machine learning model selected in accordance with the other user from among a plurality of machine learning models prepared in advance for message generation, and determine the content of the message to be sent to the other user.

6. An information processing method comprising the steps of: sequentially acquiring screen data relating to the screen content of target content having content that changes over time while the target content is being output; sequentially acquiring model output data corresponding to the screen data relating to the screen content of the target content using a machine learning model obtained by machine learning the relationship between the screen data and descriptive information that linguistically describes the content of the screen data; and executing a specified process based on the acquired model output data.

7. A program for causing a computer to execute the following processing: for target content having content that changes over time, sequentially acquire screen data relating to the screen content of the target content while the target content is being output; sequentially acquire model output data corresponding to the screen data relating to the screen content of the target content using a machine learning model obtained by machine learning the relationship between the screen data and descriptive information that linguistically describes the content of the screen data; and execute a specified process based on the acquired model output data.

Images