System, method, and computer program for generating comments corresponding to video.

Abstract

This disclosure provides a system and method for generating comments corresponding to an input video. Given an input video, it is possible to generate comments having content related to the input video. One embodiment includes a computing system comprising a processor and memory. The processor may be configured to execute a program using a portion of the memory, receive an input video in a social networking system, generate at least one predicted comment corresponding to the input video based on the video frames of the input video and the user profile of a target user, and present at least one predicted comment to the target user.

Description

Background Art

[0001] [Cross - Reference to Related Applications] This application claims priority to U.S. Application No. 18 / 301165, filed on April 14, 2023, with the title "GENERATION OF COMMENTS CORRESPONDING TO VIDEOS", the disclosure of which is hereby incorporated by reference in its entirety.

[0002] Generative machine learning algorithms can be implemented in probabilistic models that can generate new data by analyzing regularities and patterns in training datasets. In combination with language - text matching models, such generative models can generate various types of data depending on the application. Examples of such tasks include text - to - text generation, image - to - text generation, image - to - image conversion, and text - to - 3D object generation. However, recently, social media networks featuring short - form video content uploaded by users have been increasing. There are technical challenges in applying these generative machine learning models to these videos.

Summary of the Invention

Means for Solving the Problems

[0003] The present disclosure provides a system and method for generating comments corresponding to an input video. When an input video is provided, a plurality of comments related to the input video can be generated. One aspect includes a computing system comprising a processor and a memory. The processor is configured to execute a program using a portion of the memory, receive an input video in a social networking system, generate at least one predicted comment corresponding to the input video based on a user profile of a target user, and present the at least one predicted comment to the target user.

[0004] This summary is provided to present a simplified excerpt of the concept, which will be further described in the embodiments for carrying out the invention described below. This summary is not intended to identify the main or basic features of the claimed subject matter, nor to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to an implementation that solves any or all of the defects described in any part of this disclosure. [Brief explanation of the drawing]

[0005] [Figure 1] This is a schematic diagram illustrating an exemplary computing system for generating multiple comment candidates corresponding to an input video.

[0006] [Figure 2] This figure shows an input video and multiple comment candidates in a social networking system, relating to the first example of this disclosure.

[0007] [Figure 3] This figure shows an input video and multiple comment candidates in a social networking system, relating to a second example of the present disclosure.

[0008] [Figure 4] This figure shows an input video and multiple comment candidates in a social networking system, relating to a third example of the present disclosure.

[0009] [Figure 5] This diagram schematically illustrates an exemplary framework for generating multiple comment suggestions based on an input video.

[0010] [Figure 6] This is a flowchart illustrating an exemplary method for generating multiple comments corresponding to an input video.

[0011] [Figure 7] This figure schematically illustrates a non-limiting embodiment of a computing system capable of implementing one or more of the methods and processes described above. [Modes for carrying out the invention]

[0012] Traditional social media networks display pre-written suggested responses to text-based posts, allowing users to select these responses to respond more quickly to posts. However, when applied to video posts, these traditional approaches can result in irrelevant or inappropriately suggested responses to video posts. This is because traditional approaches generate suggested responses based only on the text content within the post, not the video content. As a result, users may be presented with undesirable suggested responses, potentially degrading the user experience. In light of the above challenges, this disclosure provides an example of a system and method for generating multiple comments corresponding to a given input video.

[0013] Figure 1 is a schematic diagram showing an exemplary computing system 100 for generating multiple comments corresponding to an input video 114. The computing system 100 includes a computing device 102, which further includes a processor 104 (e.g., a central processing unit, or "CPU"), an input / output (I / O) module 106, a volatile memory 108, and a non-volatile memory 110. These different components are operably coupled to one another. The non-volatile memory 110 stores a comment generation program 112, which includes instructions for various software modules described herein that are executed by the processor 104.

[0014] When executed by processor 104, instructions stored in comment generation program 112 cause processor 104 to initialize comment generation process, which includes receiving input video 114 in a social networking system, generating at least one predicted comment 132 corresponding to input video 114 based on video frames of input video 114 and user profile 116 of the target user, and presenting at least one predicted comment 132 to the target user. The at least one predicted comment 132 may be presented as text, emojis, and / or images.

[0015] The comment generation program 112 includes multiple modules, including a comment generation module 130 configured to generate comment candidates 132. The image-text matching model 136 of the comment generation module 130 receives video frames of the input video 114 as input and generates comment candidates 132 as output. The most relevant comments for the input video 114 may be identified by assigning normalized probabilities 134a to f to each comment candidate (the higher the probability, the more relevant the comment candidate). Comment candidates that exceed a predetermined probability threshold may be selected and presented to the target user as output comment candidates 138.

[0016] The image-text matching model 136 may be instantiated as a model that generates comments, or as a model that selects comments from a set of predefined comments. One exemplary instantiation of the image-text matching model 136 is a contrastive language-image pre-training (CLIP) model, which is a deep learning model that is pre-trained on a large dataset of comment-image pairs and associates images with corresponding comments. To predict comment candidates 132 for a given video frame of the input video 114, the model ranks a set of comment candidates 132 based on their degree of matching with the video frame. Using its learned knowledge of language and visual content, the image-text matching model 136 infers the comment candidates 132 that are most likely to correspond to the video frame of the input video 114 and assigns probabilities 134a to f to each comment candidate 132.

[0017] In the example in Figure 1, the first comment candidate 132a is assigned a first probability 134a of 0.95, the second comment candidate 132b is assigned a second probability 134b of 0.82, the third comment candidate 132c is assigned a third probability 134c of 0.61, the fourth comment candidate 132d is assigned a fourth probability 134d of 0.44, the fifth comment candidate 132e is assigned a fifth probability 134e of 0.38, and the sixth comment candidate 132f is assigned a sixth probability 134f of 0.25. A predetermined probability threshold may be used to output only comment candidates 138 that exceed the predetermined probability threshold. In the example in Figure 1, a probability threshold of 0.50 is used to output only the comment candidates 138 that exceed the probability threshold of 0.50, including the first comment 132a, the second comment 132b, and the third comment 132c.

[0018] The comment generation program 112 may also include a video tagging generation module 118 configured to generate video tags 120. The video tagging model 122 of the video tagging generation module 118 may receive an input video 114 and generate a video tag 120 corresponding to the input video 114. The video tag 120 may be generated based, for example, on a theme, genre, object, and / or behavior associated with the input video 114. The video tag 120 may be formatted as text or as a vector embedding. The video tagging model 122 may be instantiated as a multimodality model, or as a mixture of a convolutional neural model and a natural language processing model configured to detect high-level information about the input video 114. For example, in one possible implementation, the video tagging model 122 may accept video frames and audio of the input video 114 as input, use a speech recognition model to transcribe the audio of the input video 114, and use an optical character recognition model to detect readable text within the video frames of the input video 114. The video tagging model 122 may also accept as input one or more comments made by the target user prior to the posting of the input video 114. The transcribed audio and detected readable text may be used to generate a video tag 120. The video tag 120 is then received as input by the image-text matching model 136, which generates comment candidates 132 based on the video tag 120, the video frames of the input video 114, and the target user's user profile 116.

[0019] The comment generation program 112 may also include an embedding extraction module 124, which extracts and generates video embeddings 126 or latent spatial representations of the input video 114 using an embedding extraction model 128, which may be configured as, for example, a convolutional neural network. The extracted embeddings 126 may include feature vectors having multiple layers and attributes, each of which may have, for example, normalized values ​​between zero and one. The video embeddings 126 are then received as input by an image-text matching model 136, which generates comment candidates 132 based on the video embeddings 126, the video frames of the input video 114, and the user profile 116 of the target user.

[0020] Figure 2 shows a first exemplary use implementation of the comment generation program of Figure 1. In this example, two users interact with each other within a social networking system 200 through a user interface 202 for receiving and displaying user posts, or streams, or user posts. The user interface 202 allows users to post content, which may include videos, photos, links, and text. The content may also include actions performed by the user, such as notifications that the user has posted a new photo, commented on a post, or increased their connection with another user. The post may then be displayed to other users within a stream of content, referred to as a stream, news feed, or live feed.

[0021] In the example of FIG. 2, the first user sends the first text 204 "Here is the link" to the second user, and the second user replies with the second text 206 "then see you tmr". Then, the second user posts a video 208 of two boys dancing. The first user selects the posted video 208 and views the enlarged view 210 of the posted video 208. In response to the first user selecting the posted video 208, the comment generation program receives the video frame of the posted video 208 and the user profile of the first user as inputs, and generates comment candidates based on the video frame of the posted video 208 and the user profile of the first user. A normalization probability is assigned to each comment candidate, and a plurality of comment candidates exceeding a predetermined threshold may be selected to be presented to the first user in the enlarged view 210 as the output comment candidates 212. The comment candidates 312 include the first comment 212a "LOL" and the second comment 212b "This made my day". The comment generation program may also consider the video tags of the video 208 when generating comment candidates. For example, if the video tags of the posted video 208 indicate that the posted video 208 is related to baseball and the user profile of the first user shows an interest in baseball, comment candidates based on baseball may be generated as the output comment candidates 212. The first user selects the first comment 212a. The comment generation program receives the selection of the first comment 212a from the first user, and then displays the first comment 212a as a reply 214 to the posted video 208.

[0022] Figure 3 shows a second exemplary use implementation of the comment generation program of Figure 1. In this example, which is substantially similar to the first exemplary use implementation, two users exchange texts 304 and 306 on the user interface 302 in the social networking system 300. However, unlike the first exemplary use implementation, the comment generation program automatically generates and presents comment candidates 312 in response to detecting that the second user has posted a video 308. Comment candidates that exceed a predetermined threshold are presented to the first user on the user interface 302 as output comment candidates 312, which include a first comment 312a "LOL" and a second comment 312b "This made my day". The first user selects the first comment 312a. The comment generation program receives the selection of the first comment 312a from the first user and then displays the first comment 312a as a reply 314 to the posted video 308.

[0023] Figure 4 shows a third exemplary use implementation of the comment generation program of Figure 1. In this example, which is substantially similar to the second exemplary use implementation, two users exchange texts 404 and 406 on the user interface 402 of the social networking system 400. However, unlike the second exemplary use implementation, the comment generation program automatically generates and presents comment candidates 412 as emojis or stickers instead of text when it detects that the second user has posted a video 408. Comment candidates that exceed a predetermined threshold are presented to the first user on the user interface 402 as output comment candidates 412, and the comment candidates 312 include a first comment 412a "LOL emoji or sticker" and a second comment 412b (smile face emoji or sticker). The first user selects the first comment 412a. The comment generation program receives the selection of the first comment 412a from the first user and then displays the first comment 412a as a reply 414 to the posted video 408.

[0024] FIG. 5 is a diagram schematically showing an exemplary framework 500 for generating comment candidates 532 based on an input video 514. The framework 500 includes a video tagging model 522 that receives the input video 514 and generates a video tag 520 corresponding to the input video 514. This process may be repeated only a predetermined number of times until the video tag 520 is output. The framework 500 also includes an embedding extraction model 528 that is configured to extract and generate a video embedding 526 or a latent space representation of the input video 514. The comment generation process begins with the input video 514 being input to the video tagging model 522 and the embedding extraction model 528 to generate the video tag 520 and the video embedding 526, respectively. Then, the user profile 516 of the target user, the generated video tag 520, and the video embedding 526 are received as inputs by an image text matching model 536. By including the user profile 516 of the target user in the inputs received by the image text matching model 536, the image text matching model 536 can generate personalized comment candidates 532 for the target user of the input video 514.

[0025] The image-text matching model 536 includes an image encoder 542 for projecting video frames and video embeddings 526 of the input video 514 onto image vectors in a learned latent space, respectively. The image-text matching model 536 also includes a text encoder 540 for projecting video tags 520 and user profiles 516 onto text vectors in the same learned latent space. In such a learned latent space, the normalized probability of image-text pairs is calculated using the proximity of the image vectors and text vectors to each other. A loss calculation unit 544 may be used to calculate a loss value using the projected vectors, and the calculated loss value may be used to guide the image-text matching model 536 to generate comment candidates 532 to which normalized probabilities are assigned. A filter 546 is used to select only comment candidates that exceed a predetermined probability threshold to be presented to the target user as output comment candidates 538.

[0026] Figure 6 shows a flowchart of an exemplary method 600 for generating multiple comments corresponding to an input video. In step 602, the input video and the user profile of the target user are received.

[0027] In step 604, video tags are generated based on the input video. The video tags may be generated using a multimodality model, or a mixture of a convolutional neural model and a natural language processing model configured to detect high-level information about the input video 114. In step 606, video embeddings are generated based on the input video. The video embeddings may be latent spatial representations of the input video extracted using a convolutional neural network.

[0028] In step 608, multiple comment candidates are generated based on the input video, video embedding, video tags, and user profile. The comment candidates may be generated using an image-text matching model configured as a generative diffusion model or a CLIP model. Guidance for the image-text matching model may be provided in the form of loop feedback based on the loss calculation of the latent space vector corresponding to the input video and the generated comment candidates.

[0029] In step 610, the generated comment candidates are filtered. Filtering may be performed by selecting only the comment candidates that exceed a predetermined probability threshold. In step 612, the filtered comment candidates are presented to the target user in the social networking system.

[0030] The system and method described above are designed to help users respond to videos on social networking systems by predicting which comments they should use to reply to the video, thereby saving time, improving consistency and response rates, providing convenience, and enhancing accessibility. Multiple predicted comment candidates are generated using a trained model based on the user's user profile and the video frames of the video. Therefore, the generated comments are personalized, relevant, and appropriate to the context of the posted video, thus increasing convenience for users who do not have the time or inclination to create personalized replies to posted videos. The automated comment generation system makes the services of social media platforms more accessible to a wider range of social media users, thereby accommodating users with disabilities or language barriers who may have difficulty typing replies quickly and accurately.

[0031] In some embodiments, the methods and processes described herein may be linked to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as computer application programs or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0032] Figure 7 schematically illustrates a non-limiting embodiment of a computing system 700 capable of implementing one or more of the methods and processes described above. For example, the computing system 700 may be implemented in a head-mounted device as a controller for executing instructions to perform head pose estimation. The computing system 700 is shown in a simplified form. The computing system 700 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, game devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices.

[0033] The computing system 700 comprises a logic machine 702 and a storage machine 704. The computing system 700 may optionally include a display subsystem 706, an input subsystem 708, a communication subsystem 710, and / or other components not shown in Figure 5.

[0034] The logic machine 702 includes one or more physical devices configured to execute instructions. For example, the logic machine 702 may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logic structures. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, achieve technical effects, or achieve desired results.

[0035] The logic machine 702 may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machine 702 may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processors of the logic machine 702 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The individual components of the logic machine may optionally be distributed across two or more separate devices located remotely and / or configured for coordinated processing. Embodiments of the logic machine 702 may be virtualized and executed by computing devices connected to a remotely accessible network configured in a cloud computing setup.

[0036] The storage machine 704 includes one or more physical devices configured to hold instructions executable by a logic machine in order to implement the methods and processes described herein. When such methods and processes are implemented, the state of the storage machine 704 may be transformed, for example, to hold different data.

[0037] The storage machine 704 may include removable and / or built-in devices. The storage machine 704 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic memory (e.g., hard disk drives, floppy disk drives, tape drives, MRAM, etc.). The storage machine 704 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices.

[0038] It will be understood that the storage machine 704 includes one or more physical devices. However, alternatively, the forms of instructions described herein may be propagated by a communication medium (e.g., electromagnetic signals, optical signals, etc.) that is not retained in the physical devices for a finite period of time.

[0039] Embodiments of the logic machine 702 and the storage machine 704 may be integrated together into one or more hardware logic components. Such hardware logic components may include, for example, field-programmable gate arrays (FPGAs), programmatic and application-oriented integrated circuits (PASICs / ASICs), programmatic and application-oriented standard products (PSSPs / ASSPs), systems-on-a-chip (SOCs), and complex programmatic logic devices (CPLDs).

[0040] The terms “module,” “program,” and “engine” may be used to describe one aspect of a computing system 700 implemented to perform a particular function. In some cases, a module, program, or engine may be instantiated via a logic machine 702 that executes instructions held by a storage machine 704. It will be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated from different applications, services, code block, object, routine, API, function, etc. The terms “module,” “program,” and “engine” may include individuals or groups such as executable files, data files, libraries, drivers, scripts, database records, etc.

[0041] As used herein, “service” will be understood to be an application program that can run across multiple user sessions. A service may be available to one or more system components, programs, and / or other services. In some implementations, a service may run on one or more server computing devices.

[0042] The display subsystem 706, if included, may be used to present a visual representation of the data held by the storage machine 704. This visual representation may take the form of a graphical user interface (GUI). Since the methods and processes described herein modify the data held by the storage machine and transform the state of the storage machine, the state of the display subsystem 706 may also be transformed to visually represent the changes in the underlying data. The display subsystem 706 may include one or more display devices utilizing substantially any type of technology. Such display devices may be combined with the logic machine 702 and / or the storage machine 704 within a shared enclosure, or such display devices may be peripheral display devices.

[0043] The input subsystem 708, if included, may comprise and interact with one or more user input devices, such as a keyboard, mouse, touchscreen, or game controller. In some embodiments, the input subsystem may comprise and interact with selected natural user input (NUI) components. These components may be integrated or peripheral, and the transmission and / or processing of input actions may be handled onboard or offboard. Exemplary NUI components may include a microphone for speech and / or voice recognition, an infrared camera, color camera, stereo camera, and / or depth camera for machine vision and / or gesture recognition, a head tracker, eye tracker, accelerometer, and / or gyroscope for motion detection and / or intent recognition, and an electric field sensing component for evaluating brain activity.

[0044] The communication subsystem 710, if included, may be configured to enable communication between the computing system 700 and one or more other computing devices. The communication subsystem 710 may include wired and / or wireless communication devices compatible with one or more different communication protocols. In a non-limiting example, the communication subsystem 710 may be configured to communicate over a wireless telephone network, or over a wired or wireless local or wide-area network. In some embodiments, the communication subsystem 710 may enable the computing system 700 to send and receive messages with other devices over a network such as the Internet.

[0045] The following paragraphs provide additional support to the claims of the present application. One embodiment provides a comment generation system, the comment generation system comprising a processor and memory operablely coupled to the processor, and a comment generation program stored in the memory and, when executed by the processor, configured to receive an input video in a social networking system, generate at least one predicted comment corresponding to the input video based on video frames of the input video and a user profile of a target user, and present the at least one predicted comment to the target user. In this embodiment, additionally or alternatively, the at least one predicted comment is generated via a learning model trained on a dataset of comment-image pairs. In this embodiment, additionally or alternatively, the at least one predicted comment is selected from a plurality of comment candidates. In this embodiment, additionally or alternatively, the at least one predicted comment is generated based on one or more comments made by the target user prior to the input video. In this embodiment, additionally or alternatively, the comment generation program is further configured to generate video tags corresponding to the input video and generate at least one predicted comment based on the video tags. In this embodiment, additionally or alternatively, video tags are generated based on at least one selected from the group of themes, genres, objects, and actions. In this embodiment, additionally or alternatively, video tags are generated based on readable text detected within the video frames of the input video. In this embodiment, additionally or alternatively, the input video is labeled based on transcribed audio of the input video generated based on a speech recognition model. In this embodiment, additionally or alternatively, the comment generation program is further configured to extract embeddings from the input video and generate at least one predicted comment based on the extracted embeddings. In this embodiment, additionally or alternatively, the at least one predicted comment is at least one selected from the group of texts, emojis, and images.

[0046] Another embodiment provides a method which includes receiving an input video in a social networking system, generating at least one predictive comment corresponding to the input video based on the video frames of the input video and the user profile of a target user, and presenting the at least one predictive comment to the target user. In this embodiment, additionally or alternatively, the at least one predictive comment is generated via a learning model trained on a dataset of comment-image pairs. In this embodiment, additionally or alternatively, the at least one predictive comment is selected from a plurality of comment candidates. In this embodiment, additionally or alternatively, the at least one predictive comment is generated based on one or more comments made by the target user prior to the input video. In this embodiment, additionally or alternatively, the method further includes generating a video tag corresponding to the input video and generating at least one predictive comment based on the video tag. In this embodiment, additionally or alternatively, the video tag is generated based on at least one selected from a group of themes, genres, objects, and actions. In this embodiment, additionally or alternatively, the video tag is generated based on readable text detected within the video frames of the input video. In this embodiment, additionally or alternatively, the input video is labeled based on the transcribed audio of the input video generated based on a speech recognition model. In this embodiment, additionally or alternatively, the method further includes extracting embeddings from the input video and generating at least one predictive comment based on the extracted embeddings.

[0047] Another embodiment provides a comment generation system comprising a processor and memory operablely coupled to the processor, and a comment generation program stored in the memory and, when executed by the processor, receives an input video in a social networking system, generates a plurality of comment candidates corresponding to the input video based on the video frames of the input video, the user profile of a target user, and the embedding of the input video, presents the plurality of comment candidates to the target user, receives a selection of one of the plurality of comment candidates from the target user, and posts the selected comment candidate as a reply to the input video.

[0048] It will be understood that the configurations and / or approaches described herein are illustrative in nature and are subject to numerous modifications; therefore, these specific embodiments or examples should not be considered restrictively. The specific routines or methods described herein may represent one or more of any number of processing strategies. For this reason, the illustrated and / or described operations may be performed in parallel, in any other order, or omitted, in the order illustrated and / or described. Similarly, the order of the processes described above may be changed.

[0049] As used herein, "and / or" refers to a logical union operation, so it will be understood that A and / or B have the following truth table. [Table 1]

[0050] The subject matter of this disclosure includes novel and non-obvious combinations and subcombinations of the various processes, systems, configurations, and other features, functions, operations, and / or characteristics disclosed herein, as well as all equivalents thereof.

Claims

1. It is a comment generation system, A processor and a memory operably coupled to the processor, When stored in the memory and executed by the processor, Receive input video in a social networking system, Based on the video frames of the input video and the user profile of the target user, at least one predictive comment corresponding to the input video is generated. The system is configured to present at least one of the aforementioned predictive comments to the target user. Comment generation program and, A comment generation system that includes this feature.

2. The at least one predicted comment is generated via a learning model trained on a dataset of comment-image pairs. The comment generation system according to claim 1.

3. The aforementioned at least one predictive comment is selected from multiple comment candidates. The comment generation system according to claim 1.

4. The at least one predicted comment is generated based on one or more comments made by the target user prior to the input video. The comment generation system according to claim 1.

5. The aforementioned comment generation program further, A video tag corresponding to the aforementioned input video is generated, The system is configured to generate at least one predictive comment based on the video tag. The comment generation system according to claim 1.

6. The aforementioned video tags are generated based on at least one selected from the group of themes, genres, objects, and actions. The comment generation system according to claim 5.

7. The video tag is generated based on readable text detected within the video frames of the input video. The comment generation system according to claim 5.

8. The input video is labeled based on the transcribed audio of the input video, which is generated based on a speech recognition model. The comment generation system according to claim 7.

9. The aforementioned comment generation program further, Extract the embedded input video, The system is configured to generate at least one predictive comment based on the extracted embeddings. The comment generation system according to claim 1.

10. The aforementioned at least one prediction comment is at least one selected from the group of text, emojis, and images. The comment generation system according to claim 1.

11. It is a method, Receiving input video in a social networking system, Based on the video frames of the input video and the user profile of the target user, generate at least one predictive comment corresponding to the input video. Presenting the aforementioned at least one predictive comment to the target user, A method that includes this.

12. The at least one predicted comment is generated via a learning model trained on a dataset of comment-image pairs. The method according to claim 11.

13. The aforementioned at least one predictive comment is selected from multiple comment candidates. The method according to claim 11.

14. The at least one predicted comment is generated based on one or more comments made by the target user prior to the input video. The method according to claim 11.

15. To generate a video tag corresponding to the aforementioned input video, To generate at least one predicted comment based on the video tag, The method according to claim 11, further comprising:

16. The aforementioned video tags are generated based on at least one selected from the group of themes, genres, objects, and actions. The method according to claim 15.

17. The video tag is generated based on readable text detected within the video frames of the input video. The method according to claim 15.

18. The input video is labeled based on the transcribed audio of the input video, which is generated based on a speech recognition model. The method according to claim 17.

19. Extracting the embedded content of the aforementioned input video, To generate at least one predictive comment based on the extracted embeddings, The method according to claim 11, further comprising:

20. It is a comment generation system, A processor and a memory operably coupled to the processor, When stored in the memory and executed by the processor, Receive input video in a social networking system, Based on the video frames of the input video, the user profile of the target user, and the embedding of the input video, a plurality of comment candidates corresponding to the input video are generated. The aforementioned multiple comment candidates are presented to the target user, The target user selects one of the multiple comment candidates. A comment generation program configured to post selected comment candidates as replies to the input video, A comment generation system that includes this feature.

Images