System and method for searching and ordering personalized video

Abstract

A system and method for searching and ordering personalized videos, the example method starting from receiving a user request via a communication chat between a user and a further user. The user request includes a phrase or an emoticon. The method performs a search of a pool of personalized videos based on a user request to determine a subset of related personalized videos. The personalized video is associated with a text message. The method also includes determining a first ranking of the related personalized video. The method then continues to select a predetermined number of personalized videos from the subset of related personalized videos based on the first ranking. The method then determines a second ranking of the selected personalized videos and presents the selected personalized videos in order within the communication chat based on the second ranking. The personalized videos of the first sub-pool and the personalized videos of the second sub-pool are individually ordered.

Description

[0001] This application is a divisional application of the Chinese application with application number 202080009442.6, application date January 18, 2020, and invention name “System and method for searching and sorting personalized videos”, all contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates generally to digital image processing and more particularly to systems and methods for searching and sorting personalized videos. Background Art

[0003] Sharing media such as stickers and emojis has become a standard option in messaging applications (also referred to herein as messengers). Currently, some messengers provide users with the option to generate images and short videos and send them to other users via communication chats. Some existing messengers allow users to modify short videos before transmission. However, the modification of short videos provided by existing messengers is limited to visualization effects, filters and text. Users of current messengers cannot perform complex editing (e.g., replacing one face with another). Such video editing cannot be provided by current messengers and requires complex third-party video editing software. Summary of the invention

[0004] The purpose of this section is to introduce selected concepts in a simplified form, the details of which are described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] According to one embodiment of the present disclosure, a method for searching and sorting personalized videos is disclosed. The method may include receiving a user request by a computing device via a communication chat between a user of the computing device and another user of another computing device. The user request may include a phrase. The method may continue to perform a search of a pool of personalized videos based on the user request by the computing device to determine a subset of relevant personalized videos. Personalized videos may be associated with text messages. The search may be performed by determining that at least one word in the phrase is a synonym of at least one word in a text message associated with at least one personalized video of the pool and adding at least one personalized video to a subset of relevant personalized videos. In an example embodiment, each personalized video may be associated with one or more activity categories. In this embodiment, the search may include selecting a category from one or more activity categories based on a phrase, and determining that at least one personalized video of the pool is associated with the category. The at least one personalized video may be added to the subset of relevant personalized videos.

[0006] In another example embodiment, the search may include calculating a similarity distance between a phrase and a personalized video associated with the text message from the pool; determining whether the similarity distance does not exceed a predetermined threshold; and based on the determination that the similarity distance does not exceed the predetermined threshold, adding the personalized video to the subset of related personalized videos. In another example embodiment, the search of the pool may include selecting a personalized video from the pool based on the age of the user.

[0007] The method may also include determining, by a computing device, a first ranking of the related personalized videos. The determination of the first ranking may include calculating one or more features. The features may include a probability term that weights a function of a phrase and a text message associated with at least one related personalized video, a Jaccard similarity coefficient between terms of the phrase and terms of the text message, and a sharing rate of the related personalized videos.

[0008] Then, the method may continue to select a predetermined number of personalized videos from the relevant personalized video subset based on the first ranking by the computing device. The method may also include determining, by the computing device, a second ranking of the selected personalized videos. The determination of the second ranking may include determining a first feature vector and a second feature vector. The first feature vector may include a first global vector corresponding to a text message associated with the personalized video, a popularity measure of the personalized video, an activity category associated with the personalized video, and information indicating a personalized video previously used by the user. The first global vector may be determined based on a model of distributed word presentation. The second feature vector may include a second global vector corresponding to a phrase, an activity category that the user likes, information about the user's personal data (the information includes at least the user's age and the user's gender), and a conversation context of a communication chat. The second global vector may be determined based on a model of distributed word presentation. The determination of the second ranking may include providing the first feature vector and the second feature vector to a neural network. The neural network may be configured to output a ranking score for the personalized video.

[0009] The method may then continue with presenting, by the computing device, the selected personalized video within the communication chat.The selected personalized video may be presented in order based on the second ranking.

[0010] The pool of personalized videos may include a first sub-pool of personalized videos with pre-presented text messages and a second sub-pool of personalized videos with user-customized text messages. The selected personalized video may include at least one personalized video from the second sub-pool. The personalized videos of the first sub-pool and the personalized videos of the second sub-pool may be sorted separately. In an example embodiment, before calculating the first sorting, the personalized videos may be filtered out from the second sub-pool based on a blacklist.

[0011] According to another embodiment, a system for searching and sorting personalized videos is provided. The system may include at least one processor and a memory storing processor executable code, wherein the at least one processor may be configured to implement the operations of the method for searching and sorting personalized videos when executing the processor executable code.

[0012] According to another aspect of the present disclosure, a non-transitory processor-readable medium is provided, wherein the non-transitory processor-readable medium stores processor-readable instructions. When the processor-readable instructions are executed by a processor, the processor implements the above method for searching and sorting personalized videos.

[0013] Additional objects, advantages and novel features of the examples will be set forth in part in the following description and in part will become apparent to those skilled in the art upon examination of the following description and the accompanying drawings, or may be learned by production or operation of the embodiments. The objects and advantages of the concepts may be realized and obtained by means of the methods, instrumentalities and combinations particularly pointed out in the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.

[0015] Figure 1 is a block diagram illustrating an example environment in which systems and methods for searching and ranking personalized videos may be implemented.

[0016] Figure 2 is a block diagram illustrating an example embodiment of a computing device for implementing a method for searching and ranking personalized videos.

[0017] Figure 3 is a block diagram illustrating a system for providing personalized video according to some example embodiments of the present disclosure.

[0018] Figure 4 is a block diagram illustrating a system for searching and sorting personalized videos according to some example embodiments of the present disclosure.

[0019] Figure 5 is a block diagram illustrating a preliminary search and selection module according to some example embodiments of the present disclosure.

[0020] Figure 6 is a block diagram illustrating a quick sort module according to some example embodiments of the present disclosure.

[0021] Figure 7 is a block diagram illustrating a reordering module according to some example embodiments of the present disclosure.

[0022] Figure 8is a schematic diagram illustrating classification of personalized videos according to some example embodiments of the present disclosure.

[0023] Fig. 9 is a flow chart illustrating a process for selecting and sorting personalized videos according to some example embodiments of the present disclosure.

[0024] Fig.10 An example screen of a communication chat according to an example embodiment of the present disclosure is shown.

[0025] Fig.11 is a flow chart illustrating a method for searching and sorting personalized videos according to an example embodiment.

[0026] Fig.12 An example computer system is shown that can be used to implement a method for searching and sorting personalized videos. DETAILED DESCRIPTION

[0027] The following detailed description of the embodiments includes reference to the accompanying drawings that form a part of the detailed description. The methods described in this section are not prior art to the claims and are not admitted to be prior art by inclusion in this section. The accompanying drawings show descriptions according to exemplary embodiments. These exemplary embodiments, which are also referred to as "examples" in this article, are described in sufficient detail to enable those skilled in the art to practice this subject matter. Without departing from the scope of the claimed protection, the embodiments may be combined, other embodiments may be utilized, or structural, logical and operational changes may be made. Therefore, the following detailed description should not be considered restrictive, and the scope is limited by the attached claims and their equivalents.

[0028] For the purposes of this patent document, the terms "or" and "and" shall mean "and / or" unless otherwise stated or clearly indicated otherwise in the context of their use. The terms "a" and "an" shall mean "one or more" unless otherwise stated or where the use of "one or more" is clearly inappropriate. The terms "comprise," "comprising," "include," and "including" are interchangeable and are not intended to be limiting. For example, the term "comprising" shall be interpreted to mean "including, but not limited to."

[0029] The present disclosure relates to methods and systems for searching and sorting personalized videos. The embodiments provided in the present disclosure solve at least some of the problems of the prior art. The present disclosure can be designed to be implemented in real time on a mobile device such as a smart phone, tablet computer or phone, but the embodiments can be extended to network services or cloud-based resources. The methods described in this article can be implemented by software running on a computer system and / or by hardware utilizing a combination of microprocessors or other specially designed application-specific integrated circuits (ASICs), programmable logic devices or any combination thereof. Specifically, the methods described in this article can be implemented by a series of computer executable instructions residing on a non-transitory storage medium (e.g., a disk drive or a computer-readable medium).

[0030] Some embodiments of the present disclosure may allow for the generation of personalized videos in real time on a user computing device such as a smartphone. The personalized video may be generated based on a pre-generated video template. The video template may include a sequence of frame images and preset text parameters for text animation. The video template may also include a sequence of facial region parameters defining the position of facial regions in the frame images, and a sequence of facial landmark parameters defining the position of facial landmarks in the frame images. Each facial landmark parameter may correspond to a facial expression. The frame images may be generated based on an animated video or a live action video.

[0031] The facial landmark parameters may be generated based on another live-action video representing the actor's face (also referred to as facesync, as described in more detail below), an animated video, an audio file, text, or manually. The video template may include a sequence of animated object images. The video template may also include a soundtrack. As used herein, a facesync actor is a person whose facial landmark parameters are being used, an actor is another person whose body is being used in the video template and whose skin may be recolored, and a user is a person whose face is imaged to generate a personalized video. Thus, in some embodiments, a personalized video may include the user's face modified to have the facial expressions of the facesync actor, and include the actor's body taken from the video template and recolored to match the user's facial color.

[0032] Pre-generated video templates can be stored remotely in cloud-based computing resources and can be downloaded by users of computing devices (such as smart phones). Users of computing devices can capture images of faces through computing devices or select images of faces from camera rolls, from prepared image collections, or via network links. In some embodiments, images can include faces of animals instead of people, or can be in the form of drawings. Based on the image of the face and one of the pre-generated video templates, the computing device can also generate personalized videos. Users can send personalized videos to another user of another computing device via communication chat, share on social media, download to local storage devices of computing devices, or upload to cloud storage devices or video sharing services.

[0033] The computing device may also add text to the generated personalized video. The computing device may generate a configuration file based on preset text parameters. The computing device may receive input text to be added to the personalized video from the user. The computing device may render the input text on each frame according to the text parameters in the configuration file. As a result, the input text may appear as an animation in the personalized video. A user may send a personalized video with preset text or a personalized video with customized text to another user of another computing device via a communication chat.

[0034] In some embodiments, the generation of personalized videos with customized text messages can be integrated into the messenger. Users can search for personalized videos using keywords, phrases, or emoticons. The messenger may include a system for searching and sorting personalized videos. The system for searching and sorting personalized videos may allow a pool of personalized videos to be searched to determine the most relevant personalized videos. The pool may include personalized videos with preset text and personalized videos with customized text. The search for relevant personalized videos may be based on keywords or phrases, the context of a user's communication chat with another user of another computing device, user preferences, popularity metrics of personalized videos, etc.

[0035] According to one embodiment of the present disclosure, an example method for searching and sorting personalized videos may include receiving a user request via a communication chat between a user of a computing device and another user of another computing device. The user request may include a phrase. The method may continue to perform a search of a pool of personalized videos based on the user request to determine a subset of relevant personalized videos. The personalized video may be associated with a text message. The method may include determining a first ranking of relevant personalized videos, and selecting a predetermined number of personalized videos from the subset of relevant personalized videos based on the first ranking. Then, the method may continue to determine a second ranking of the selected personalized videos, and present the selected personalized videos in order within the communication chat based on the second ranking.

[0036] With reference now to the accompanying drawings, exemplary embodiments are described. The accompanying drawings are schematic diagrams of idealized exemplary embodiments. Therefore, the exemplary embodiments discussed herein should not be construed as being limited to the specific descriptions presented herein; rather, as will be apparent to those skilled in the art, these exemplary embodiments may include deviations and differences from the descriptions presented herein.

[0037] Figure 1 1 is an example environment 100 in which a method for searching and sorting personalized videos may be implemented. Environment 100 may include computing device 105, user 102, computing device 110, user 104, network 120, and messenger service system 130. Computing device 105 and computing device 110 may refer to mobile devices such as phones, smart phones, or tablets. In other embodiments, however, computing device 110 may refer to a personal computer, a laptop, a netbook, a set-top box, a television device, a multimedia device, a personal digital assistant, a game console, an entertainment system, an infotainment system, a car computer, or any other computing device.

[0038] Computing device 105 and computing device 110 may be communicatively connected to messenger service system 130 via network 120. Messenger service system 130 may be implemented as a cloud-based computing resource. Messenger service system 130 may include computing resources (hardware and software) available at a remote location and accessible via a network (e.g., the Internet). Cloud-based computing resources may be shared by multiple users and may be dynamically reallocated based on demand. Cloud-based computing resources may include one or more server farms / clusters, which include a collection of computer servers that may be co-located with network switches and / or routers.

[0039] Network 120 may include any wired network, wireless network, or optical network (e.g., including the Internet, an intranet, a local area network (LAN), a personal area network (PAN), a wide area network (WAN), a virtual private network (VPN), a cellular telephone network (e.g., Global System for Mobile Communications (GSM)), etc.).

[0040] In some embodiments of the present disclosure, computing device 105 may be configured to initiate a communication chat between user 102 and user 104 of computing device 110. User 102 and user 104 may exchange text messages and videos during the communication chat. The video may include a personalized video. The personalized video may be generated based on a pre-generated video template stored in computing device 105 or computing device 110. In some embodiments, the pre-generated video template may be stored in messenger service system 130 and downloaded to computing device 105 or computing device 110 as needed.

[0041] The messenger service system 130 may include a system 140 for preprocessing videos. The system 140 may generate video templates based on animated videos or live action videos. The messenger service system 130 may include a video template database 145 for storing video templates. The video templates may be downloaded to the computing device 105 or the computing device 110. The video template database may also store statistical information of the downloads of the templates to determine the popularity metrics of the video templates. The popularity metrics may be divided into multiple categories according to the age of the user, the gender of the user, the geographic region, etc.

[0042] The messenger service system 130 may also be configured to store user profiles 135. The user profile 135 may include an image of the face of the user 102, an image of the face of the user 104, and an image of the face of other people. The image of the face may be downloaded to the computing device 105 or the computing device 110 on demand and based on permission. In addition, the image of the face of the user 102 may be generated using the computing device 105 and stored in the local memory of the computing device 105. The image of the face may be generated based on other images stored in the computing device 105. The computing device 105 may also use the image of the face to generate a personalized video based on a pre-generated video template. Similarly, the computing device 110 may be used to generate an image of the face of the user 104. The image of the face of the user 104 may be used to generate a personalized video on the computing device 110. In other embodiments, the image of the face of the user 102 and the image of the face of the user 104 may be used to generate a personalized video on the computing device 105 or the computing device 110.

[0043] In some embodiments, the pre-generated video template may include a preset default text message. The pre-generated video template may also include preset text parameters, which are used to animate the preset default text message in the personalized video generated based on the pre-generated video template. The computer device may be configured to play a personalized video that characterizes the preset text message animated based on the preset text parameters. During the playback of the personalized video, the computing device 105 may provide options for changing the preset default text message and parameters for displaying the text message in the personalized video. The computing device 105 may dynamically change the preset default text message in the personalized video while playing the personalized video. The user 105 may also send the personalized video with the changed text message to the user 104 of the computing device 110 via the communication chat.

[0044] In an exemplary embodiment, if in the communication chat, user 104 uses a language different from the language of user 102, the text in the text message can be automatically translated. Therefore, in the communication chat, user 104 can see a personalized video with the text message displayed in the language used by user 104.

[0045] In some embodiments, user 102 may search for personalized videos to send in a communication chat. User 102 may type a keyword or phrase. Computing device 105 may perform a search for relevant personalized videos with customized text and present a list of relevant personalized videos for selection by user 102. The search may be based on the keyword or phrase, the communication context in the communication chat, the user's age, the user's gender, a geographic region, a popularity metric for the personalized video, and the like.

[0046] Figure 2 is a block diagram illustrating an example embodiment of a computing device 105 (or computing device 110) for implementing a method for searching and sorting personalized videos. Figure 2 In the example shown, computing device 110 includes both hardware components and software components. Specifically, computing device 110 includes a camera 205 or any other image capture device or scanner for acquiring digital images. Computing device 110 may also include a processor module 210 and a memory storage device 215 for storing software components and processor-readable (machine-readable) instructions or codes that, when executed by processor module 210, cause computing device 105 to perform at least some steps of the method for searching and sorting personalized videos as described herein. Computing device 105 may include a graphics display system 230 and a communication module 240. In other embodiments, computing device 105 may include additional or different components. Furthermore, computing device 105 may include a processor module 210 and a memory storage device 215 for storing software components and processor-readable (machine-readable) instructions or codes that, when executed by processor module 210, cause computing device 105 to perform at least some steps of the method for searching and sorting personalized videos as described herein. Computing device 105 may include a graphics display system 230 and a communication module 240. In other embodiments, computing device 105 may include additional or different components. Furthermore, computing device 105 may include a processor module 210 that performs the processing associated with Figure 2 Fewer components having similar or equivalent functions as depicted.

[0047] The computing device 110 may also include a messenger 220 for initiating a communication chat with another computing device (e.g., computing device 110), a system 250 for generating personalized videos, and a system 260 for searching and sorting personalized videos. Figure 3 The system 250 is described in more detail below. Figure 4 The system 260 is described in more detail. The messenger 220, the system 250, and the system 260 may be implemented as software components and processor-readable (machine-readable) instructions or codes stored in the memory storage device 215, which, when executed by the processor module 210, causes the computing device 105 to perform at least some steps of the method for providing communication chat, generating personalized videos, and searching and sorting personalized videos as described herein.

[0048] In some embodiments, the system 250 for generating personalized videos and the system 260 for searching and sorting personalized videos may be integrated in the messenger 220. The user interface of the messenger 220, the system 250 for generating personalized videos, and the system 260 for searching and sorting personalized videos may be provided via the graphic display system 230. The communication chat may be initiated via the communication module 240 and the network 120. The communication module 240 may include a GSM module, a WiFi module, a Bluetooth module, a TM Modules, etc.

[0049] Figure 3 is a block diagram of a system 250 for generating personalized videos according to some example embodiments of the present disclosure. The system 250 may include a user interface 305, a facial image capture module 310, a template database 320, a personalized video generation module 330, and a pool 325 of personalized videos (e.g., reels).

[0050] The video template database 320 may store video templates for generating personalized videos. The video template 320 may include the video templates obtained from the messenger service system 130 ( Figure 1 The video template may include a previously recorded video characterizing one or more actors. The video may include a 2D video or a 3D scene. The video template may be pre-processed to segment the actor's face (also referred to as a target face) and the background in each frame, and identify a set of parameters that can be used to further insert a source face instead of the actor's face (target face). The set of parameters may include facial texture, facial expression parameters, facial color, facial identity parameters, the position and angle of the face, etc. The set of parameters may also include a list of manipulations and operations that can be performed on the actor's face (such as, replacement of the actor's face in a photo-realistic manner).

[0051] The facial image capture module 310 may receive an image of a person and generate a facial image of the person. The facial image of the person may be used as a source face to replace a target face in a video stored in the video template database 320. The image of the person may be captured by the camera 205 of the computing device 105. The image of the person may include an image stored in the memory storage device 215 of the computing device 105.

[0052] The personalized video generation module 330 may generate a personalized video from one or more pre-generated video templates stored in the database 320 based on an image of a source face. The module 330 may replace the face of an actor in a pre-generated video with the source face while maintaining the facial expression of the actor's face. The module 330 may replace the facial texture, facial color, and facial identity of the actor with the facial texture, facial color, and facial identity of the source face. The module 330 may also add an image of glasses to the eye area of ​​the source face in the personalized video. Similarly, the module 330 may add an image of a helmet (e.g., a cap, a bowler hat, a helmet, etc.) to the head of the source face in the personalized video. The images of the glasses and helmet may be pre-stored in the user's computing device 105, or the images of the glasses and helmet may be generated. A DNN may be used to generate images of glasses and headwear. The module 330 may also apply shadows or colors to the source face in the personalized video.

[0053] The pool 325 of personalized videos may include videos generated based on video templates from the video template database 320. The pool 325 may include personalized videos with preset text and personalized videos with customized text. The personalized videos are also referred to herein as volumes.

[0054] Figure 4 4 is a block diagram illustrating a system 260 for searching and sorting personalized videos according to some example embodiments of the present disclosure. At block 405, a user request may be received via a communication chat between a user of a computing device and another user of another computing device. The user request may include a phrase. A preliminary search and selection module 410 may perform a search of a pool of personalized videos based on the user request to determine a subset 415 of relevant personalized videos. The personalized video may be associated with a text message. In an example embodiment, the personalized video may be selected based on a list of personalized videos stored in a video template database 145 ( Figure 1 as shown) or video template database 320 ( Figure 3 The search is performed using templates in (shown in FIG. 1 ). The video templates may be personalized and presented to the user. Thus, the preliminary search and selection module 410 may provide an unordered pool of relevant personalized videos.

[0055] After performing the preliminary search, the quick sort module 420 can determine a first order of relevant personalized videos. After determining the first order, a predetermined number of personalized videos 415 can be selected from a subset of relevant personalized videos (volumes) based on the first order. The selected predetermined number of personalized videos are displayed as a pool 425 of selected personalized videos (volumes). The reorder module 430 can determine a second order of the selected predetermined number of personalized videos. After determining the second order, an ordered pool 435 of selected personalized videos (volumes) can be presented in the communication chat. The selected personalized videos can be presented in the pool in an order determined based on the second order.

[0056] Figure 5 is a block diagram illustrating a preliminary search and selection module 410 according to some example embodiments of the present disclosure. The preliminary search and selection module 410 may be configured to perform word matching 505. Many words in a language may have different forms (e.g., "amazing" and "amazed"), but their meanings are similar in terms of search requests. The preliminary search and selection module 410 may have a search engine that is configured to reduce inflected words to their base or root form. To this end, a stemming technique that implies the use of a lookup table for converting each word to a base form may be used. This approach allows for optimal control of algorithm behavior and easily handles extreme cases. For better search performance, synonyms may be used to expand search requests (i.e., user requests). A table with a set of synonymous phrases (all phrases have similar meanings in each set) may be provided. If one of the phrases is present in the search request, the search request may be expanded by replacing the phrase with any other phrase from the same set.

[0057] In order to make the user experience smooth, it may be very important to quickly find the appropriate personalized video. A data structure called an inverted index can be used. The inverted index data structure can efficiently find personalized videos with specific words in the description. Moreover, prefix matching can be applied to the last word in the user request.

[0058] All personalized videos found during this step can be added to Figure 4 The subset 415 of relevant personalized videos is shown in the ranked pool.

[0059] Another important issue is to avoid toxic content in customized personalized videos. To this end, fuzzy matching with blacklist words can be used. For each word from the search request, a weighted edit distance (Levenshtein distance) can be calculated with each word from a predetermined blacklist. A dynamic programming algorithm can be used to calculate the weighted edit distance:

[0060] dp[i][j]=min(dp[i][j-1]+cost(b j ), dp[i-1][j]+cost(a i ), dp[i-1][j-1]+sim(a i , b j ))

[0061] if a[i]==b[j]:dp[i][j]=min(dp[i][j], dp[i-1][j-1])

[0062] Where cost(ai) is the insertion cost of a character, sim(a i ,b j ) is the similarity distance between characters.

[0063] Thus, for example, the weighted edit distance may be lower for the words "bigger" and "bigger" because the similarity distance sim("1", "i") is smaller. If the distance is less than a certain predetermined threshold, the word may be considered toxic and a customized personalized video with the word may not be displayed.

[0064] The preliminary search and selection module 410 may be configured to perform activity category matching 510. All personalized videos may be grouped into a plurality of activity categories. Activity categories may be defined, for example, by user activities such as happy, upset, angry, love, greeting, celebration, food, surprise, question, neutral, etc. Activity category matching 510 aims to predict the activity category of the user request based on a neural network classifier and add personalized videos from that category to a pool of related personalized videos.

[0065] A neural network classifier can be composed of the following components: a unigram language model and a multi-layer recurrent neural network. The unigram language model can be used for data preprocessing. The unigram language model maximizes the following marginal likelihood:

[0066]

[0067] Where L is the likelihood, P is the probability of the subword sequence, and S is the set of segmentation candidates. The data is then fed into a multi-layer recurrent neural network with instance normalization and leaky rectifier (ReLU) as activation functions. A self-attention mechanism can be used as the last layer of the neural network. To train the neural network, a dataset of 10,000 user requests with specified categories is manually created. For better convergence, a supervised lookahead optimizer with categorical cross entropy loss is used to train the model.

[0068] In one example embodiment, to improve the ranking of customized personalized videos, ten categories describing user activities (happy, upset, angry, love, greeting, celebration, food, surprise, question, and neutral) may be identified. The search engine may refer each search request to one of these activities. To identify the activities, natural language understanding techniques may be used. For each activity type, information about the share rate of customized personalized videos may be collected. Ranking may be improved by optimizing customized personalized videos with high share rates for user requests with certain activities.

[0069] The preliminary search and selection module 410 can be configured to perform sentence embedding matching 515. In any language, there are many words with similar meanings. The purpose of this step is to provide relevant results even if an exact word match is not found. The sentence embedding is calculated as the average of the global vector (GloVe) embeddings for all words from the user request. Moreover, for each description of the personalized video, the embedding can be pre-calculated in a similar manner.

[0070] For any user request, sentence embedding matching 515 can search for personalized videos with the closest embedding. This can be done in a high-dimensional space using Locality-Sensitive Hashing. All personalized videos with a cosine similarity greater than 0.7 can be added to the Figure 4 A subset 415 of related personalized videos is shown.

[0071] In an example embodiment, a user may use an emoji or a combination of an emoji and other symbols / words as a search request to search for personalized videos. The GloVe model may be trained on a text corpus similar to that used by people in communication chats, so the model may include multiple emojis. The trained model may provide a vector representation for each token in the corpus. For each emoji, an embedding vector and cosine distance may be used to find the word with the closest embedding. Thus, the emoji may be replaced with a word that describes the emoji.

[0072] In another example embodiment, the context of the chat session can significantly improve the quality of the search. For example, if the user is chatting about football, personalized videos related to football can be promoted higher in the search results.

[0073] A neural network-based approach can be used to extract content information from chat sessions. The algorithm can be based on a transformer structure with multi-head self-attention. A powerful pre-trained model allows extracting meaningful embedding vectors from users' conversations. Based on the embedding, transfer learning techniques can be used to obtain sufficient results for downstream tasks. Before feeding the text to the transformer, it may be necessary to provide position embeddings for distinguishing words at different positions. For this purpose, sine and cosine functions with different frequencies can be used. The following formula describes how to generate position embeddings:

[0074] PosEmb(pos,2i)=sin(pos / 10000 2i / d )

[0075] PosEmb(pos,2i+1)=cos(pos / 10000 2i / d )

[0076] where pos is the token number, d is the position embedding dimension, and i is the coordinate in the embedding.

[0077] Figure 6 6 is a block diagram illustrating a quick sorting module 420 according to some example embodiments of the present disclosure. The quick sorting module 420 may be configured to perform personalized video feature extraction 605. To implement quick sorting, the following features may be extracted: probability weighted sum BM11, BM15, and BM25, dot product of GloVe embeddings for user queries and descriptions of personalized videos, global sharing rates of personalized videos, specific sharing rates of personalized video activities, Jaccard coefficients of common terms of user queries and descriptions of personalized videos, etc. The following formula may be used to determine the Jaccard coefficient:

[0078]

[0079] The quick sort module 420 can be configured to perform a tree-based gradient boosting sort 610 (i.e., a gradient boosting tree can be used for quick sorting). After this step, a predetermined number of the best personalized videos are selected for a re-ranking step. Quick sorting allows for better quality results to be obtained with high efficiency.

[0080] Figure 7 705 is a block diagram illustrating a reordering module 430 according to some example embodiments of the present disclosure. The reordering module 430 may be configured to perform video feature extraction 705. Specifically, the following features may be extracted for the selected personalized video: GloVe embedding, popularity metrics, activity categories, information about whether the user has seen the personalized video before, etc. These features may be stacked into an embedding vector EmbC associated with the personalized video.

[0081] The re-ranking module 430 may be configured to perform user data feature extraction 710. The following features may be extracted from the user data: query embedding, user's favorite categories, and basic user information (such as gender, age, etc.). In addition, session context may also be used as a feature. These features may be stacked into an embedding vector EmbU associated with the user.

[0082] The re-ranking module 430 may be configured to perform a ranking score calculation 715. EmbC and EmbU may be fed into a fully connected neural network having a perceptron in the last layer that predicts the ranking score. In addition, the re-ranking module 430 may perform a video order 720 to rank the personalized videos based on the ranking score. Specifically, personalized videos with higher scores may appear higher in the search results.

[0083] To train the two ranking models, a dataset of at least 10,000 user requests (same as for activity category prediction) can be used. In this case, for each user request, five relevant personalized videos can be assigned. The model can be optimized for each pair of user request and relevant personalized video to increase the ranking score. Moreover, for each user request, five random and five irrelevant personalized videos can be selected. In addition, the model can be trained for each pair of user request and one of these personalized videos to reduce the ranking score.

[0084] Figure 8 800 is a schematic diagram showing the classification of personalized videos according to some example embodiments of the present disclosure. Two different types of personalized videos need to be sorted: personalized videos with pre-rendered text and customized personalized videos (with text given by the user). In order to get the final order, the personalized videos can be sorted separately within each type, and the results can be merged according to the logic described below.

[0085] Personalized videos can be divided into six categories in the category section, and users can select categories and view the best reels corresponding to user intent. The six categories for personalized videos may include: Featured, Greetings, Love, Happy, Upset, and Celebration. In each category, personalized videos can be displayed to users. Users can be divided into four age groups: under 18, 18 to 30, over 30, and not applicable (N / A), and three gender groups: male, female, and not applicable (N / A). There can be twelve groups of users in general. Moreover, two different types of content can be used, namely personalized videos that previously existed in the application and for which performance metrics have been calculated, and new personalized videos. Specifically, if the search results are sorted only according to the relevance of personalized videos, users can get the same search results in each session. In order to make the user more engaged, the search results can be changed by adding new personalized videos that the user has not seen before. Adding new personalized videos may not affect the relevance of the search results.

[0086] Reference Figure 8, List A specifies the best personalized videos with determined performance metrics for each age group, gender group, and category to which the volume belongs. List B specifies new personalized videos for different user groups for each category where there is no information about performance metrics and performance assumptions. List B may have friend personalized videos and customized personalized videos. List C specifies personalized videos with determined performance metrics for each age group and gender group. Customized personalized videos allow users to add any text they want to personalized videos. When the user enters any text, the text can be placed on these personalized videos, and the personalized videos can appear in the quick search section. The text can have preset animations and styles, and the narration of the personalized videos can be universal. Some personalized videos may have customized versions and pre-rendered versions. Customized personalized videos may have default text so that personalized videos can be displayed in the category section. Search metadata information may include search descriptions and tags describing usage scenarios related to personalized videos. This data may exist in all personalized videos.

[0087] In one example embodiment, personalized videos may be selected from List A and List B for each category by an algorithm. For each group and each category, the following algorithm may be applied to a predetermined number of reels and their order:

[0088] The personalized videos at positions 1, 2, 7, ... 24 are selected from list A according to the cumulative sharing rate of the specific group.

[0089] The personalized videos on positions 3, 4, 5, 6 are taken from list B and are the same for all groups. This step can help introduce new personalized videos and get statistics.

[0090] If some personalized videos from list B are friend personalized videos and this feature is disabled in the current communication chat, the personalized videos from list B may be replaced with personalized videos from list A according to their order.

[0091] Sometimes List B may have a smaller size for certain categories, but similar logic may apply to List B.

[0092] In addition, the recent category may be personalized for a particular user and may include recently shared personalized videos. The number of personalized videos in the recent category may not exceed a predetermined number. If the number of recently shared personalized videos is less than 8, the remaining positions in the category may be filled with personalized videos from the featured category. If some of these personalized videos are customized, a version of the personalized video with default text may be used.

[0093] Typically, ranking can be used to provide a quick and easy way to get the best personalized videos for popular user requests. To this end, user metrics for personalized videos can be tracked, and only high-performing personalized videos can be kept. User metrics for new personalized videos can be evaluated. In addition, personalized lists for different user groups can be provided based on personalized video statistics. In addition, user intent prediction based on machine learning can be utilized.

[0094] Fig. 9 900 is a flowchart illustrating a process for selecting and sorting personalized videos according to some example embodiments of the present disclosure. The selection and sorting of personalized videos that provide search results 940 for L volumes may include the following steps. In a first step, the best N pre-rendered volumes 910 may include relevant (if any) pre-rendered personalized videos (volumes) from list C. Pre-rendered volumes 905 from list C may be added only if the pre-rendered text on these volumes is relevant to the user request. In a second step, the best M customized volumes 930 may include relevant customized volumes 905 selected from customized volumes 915 from list B, customized volumes 920 from list A, and customized volumes 925 from list C, which are sorted according to their relevance to the search query. Custom volumes that are not allowed due to technical limitations are filtered out (check whether the user text can be suitable for personalized videos). The (LNM) remaining search results 935 of pre-rendered volumes may include volumes that are less relevant to the pre-rendered volumes 905 from list C.

[0095] If the search query includes at least one phrase from a predetermined blacklist of customized personalized videos, the customized personalized video may not be presented for the search query. The blacklist may include only offensive speech / muffled sounds, and in some embodiments may allow more standard profanity.

[0096] For some search queries, both customized and pre-rendered versions of the same personalized video may be selected for the search report. To avoid showing two similar personalized videos, only one of these personalized videos may be selected in the final search results. If the search query exactly matches (or prefixes) the pre-rendered personalized video text, the pre-rendered version of the personalized video may be displayed. Otherwise, the customized version of the personalized video may be presented.

[0097] Fig.10 An example screen 1000 of a communication chat according to an example embodiment of the present disclosure is shown. A quick search icon 1015 may be presented to the user to enable the user to select one of the videos 1010. Clicking the quick search icon 1015 may open a sticker selector on the search tab 1020. Based on the activity category determined for the search query 1005, the user's face on the personalized video 1010 may have an emotional reenactment and some visual elements surrounding it.

[0098] Fig.11 1 is a flow chart illustrating a method for searching and ranking personalized videos according to an example embodiment. The method 1100 may be performed by a computing device 105. The method 1100 may begin by receiving a user request via a communication chat between a user and another user at step 1105. The user request may include a phrase. The method 1100 may continue to perform a search of a pool of personalized videos based on the user request at step 1110. The search of the pool of personalized videos may be performed to determine a subset of relevant personalized videos. The personalized videos may be associated with a text message. The method 1100 may also include determining a first ranking of relevant personalized videos at step 1115. The method 1100 may continue to select a predetermined number of personalized videos from the subset of relevant personalized videos at step 1120. The predetermined number of personalized videos may be selected based on the first ranking. The method 1100 may also include determining a second ranking of the selected personalized videos at step 1125. At step 1130, the selected personalized videos may be presented in order within the communication chat based on the second ranking.

[0099] Fig.12 An example computing system 1200 is shown that can be used to implement the methods described herein. Computing system 1200 can be implemented in environments like computing devices 105 and 110, messenger service system 130, messenger 220, system for generating personalized videos 250, and system for searching and ranking personalized videos 260.

[0100] like Fig.12 As shown, the hardware components of the computing system 1200 may include one or more processors 1210 and a memory 1220. The memory 1220 stores, in part, instructions and data for execution by the processor 1210. The memory 1220 may store executable code when the system 1200 is running. The system 1200 may also include an optional mass storage device 1230, an optional portable storage medium drive 1240, one or more optional output devices 1250, one or more optional input devices 1260, an optional network interface 1270, and one or more optional peripheral devices 1280. The computing system 1200 may also include one or more software components 1295 (e.g., a software component that may implement the method for providing personalized video as described herein).

[0101] Fig.12The components shown are depicted as being connected via a single bus 1290. The components may be connected via one or more data transfer devices or data networks. The processor 1210 and memory 1220 may be connected via a local microprocessor bus, and the mass storage device 1230, peripheral device 1280, portable storage device 1240, and network interface 1270 may be connected via one or more input / output (I / O) buses.

[0102] Mass storage 1230, which may be implemented with a magnetic disk drive, solid-state disk drive, or optical disk drive, is a non-volatile storage device for storing data and instructions for use by processor 1210. Mass storage 1230 may store system software (e.g., software component 1295) for implementing the embodiments described herein.

[0103] The portable storage media drive 1240 operates in conjunction with a portable non-volatile storage medium, such as a compact disk (CD) or digital video disk (DVD), to input and output data and code to and from the computing system 1200. System software (e.g., software component 1295) for implementing the embodiments described herein may be stored on such portable media and input to the computing system 1200 via the portable storage media drive 1240.

[0104] Optional input device 1260 provides a portion of the user interface. Input device 1260 may include an alphanumeric keypad (e.g., a keyboard) or a pointing device (e.g., a mouse, trackball, stylus, or cursor direction keys) for entering alphanumeric and other information. Input device 1260 may also include a camera or scanner. In addition, Fig.12 The illustrated system 1200 includes an optional output device 1250. Suitable output devices include speakers, printers, network interfaces, and monitors.

[0105] The network interface 1270 may be used to communicate with external devices, external computing devices, servers, and networking systems via one or more communication networks, such as one or more wired networks, wireless networks, or optical networks, including, for example, the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a cellular phone network, a Bluetooth radio, and a radio frequency network based on IEEE802.11, etc. The network interface 1270 may be a network interface card (such as an Ethernet card, an optical transceiver, a radio frequency transceiver), or any other type of device capable of sending and receiving information. The optional peripheral devices 1280 may include any type of computer support device to add additional functionality to the computer system.

[0106] The components included in the computing system 1200 are intended to represent a large class of computer components. Thus, the computing system 1200 may be a server, a personal computer, a handheld computing device, a phone, a mobile computing device, a workstation, a minicomputer, a mainframe computer, a network node, or any other computing device. The computing system 1200 may also include different bus configurations, networking platforms, multi-processor platforms, etc. Various operating systems (OS) may be used, including UNIX, Linux, Windows, MacintoshOS, PalmOS, IOS, Android, and other suitable operating systems.

[0107] Some of the above functions may consist of instructions stored on a storage medium (e.g., a computer-readable medium or a processor-readable medium). The instructions may be retrieved and executed by a processor. Some examples of storage media are storage devices, tapes, disks, etc. The instructions are operable when executed by the processor to direct the processor to operate in accordance with the present invention. Those skilled in the art are familiar with instructions, processors, and storage media.

[0108] It is noteworthy that any hardware platform suitable for performing the processing described herein is suitable for the present invention. As used herein, the terms "computer-readable storage medium" and "computer-readable storage medium" refer to any medium that participates in providing instructions to a processor for execution. Such media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks (such as fixed disks). Volatile media include dynamic memory (such as system random access memory (RAM)).

[0109] Transmission media include coaxial cables, copper wires, and optical fibers, etc., including wires that comprise an embodiment of a bus. Transmission media may also take the form of sound waves or light waves (such as those generated during radio frequency (RF) and infrared (IR) data communications). Common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD read-only memory (ROM) disks, DVDs, any other optical media, any other physical media with patterns of marks or holes, RAM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), any other memory chip or cassette, carrier waves, or any other medium from which a computer can read.

[0110] Various forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution. The bus carries the data to a system RAM from which the processor retrieves and executes the instructions. The instructions received by the system processor may optionally be stored on a fixed disk before or after execution by the processor.

[0111] Thus, a method and system for searching and ranking personalized videos has been described. Although the embodiments have been described with reference to specific example embodiments, it will be apparent that various modifications and changes may be made to these example embodiments without departing from the broader spirit and scope of the present application. Accordingly, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A method comprising: receiving, by a computing device via a communication chat between a user of the computing device and another user of another computing device, a category selected by the user from a set of predetermined categories; selecting, by the computing device, a first subset of videos from a pool of videos, a first video in the first subset of videos being associated with the category and having a calculated value of a performance metric; selecting, by the computing device, a second subset of videos from the pool of videos, second videos in the second subset of videos being associated with the category and having an unknown value of the performance metric; Generating, by the computing device, a third video set based on a predetermined rule, the third video set including at least one video from the first video subset and at least one video from the second video subset; and The third set of videos is presented by the computing device via the communication chat.

2. The method according to claim 1, wherein: The categories are associated with emotional expressions.

3. The method according to claim 1, wherein: The performance metric includes a popularity metric.

4. The method according to claim 1, wherein: The performance metric is determined based on a group to which the user belongs.

5. The method according to claim 4, wherein: The group comprises one of: an age group and a gender group.

6. The method according to claim 1, wherein: The second video is modified by the additional user.

7. The method according to claim 6, wherein: The second video includes text modified by the additional user.

8. The method according to claim 1, wherein: The first subset of videos is ordered based on an ordering associated with videos in the pool of videos; and The first subset of videos is selected based on the ranking.

9. The method according to claim 8, wherein: The ranking is based on the performance metric.

10. The method according to claim 1, further comprising, before generating the third video set: determining that the number of videos in the second video subset is less than a predetermined number; and In response to the determination, an additional video from the pool of videos is added to the second subset of videos, the additional video being associated with an additional category from the set of predetermined categories.

11. A computing device comprising: processor; as well as A memory storing instructions, which, when executed by the processor, configure the computing device to: receiving, by the computing device via a communication chat between the user of the computing device and another user of another computing device, a category selected by the user from a set of predetermined categories; selecting, by the computing device, a first subset of videos from a pool of videos, a first video in the first subset of videos being associated with the category and having a calculated value of a performance metric; selecting, by the computing device, a second subset of videos from the pool of videos, second videos in the second subset of videos being associated with the category and having an unknown value of the performance metric; Generating, by the computing device, a third video set based on a predetermined rule, the third video set including at least one video from the first video subset and at least one video from the second video subset; and The third set of videos is presented by the computing device via the communication chat.

12. The computing device of claim 11, wherein: The categories are associated with emotional expressions.

13. The computing device of claim 11, wherein: The performance metric includes a popularity metric.

14. The computing device of claim 11, wherein: The performance metric is determined based on a group to which the user belongs.

15. The computing device of claim 14, wherein: The group comprises one of: an age group and a gender group.

16. The computing device of claim 11, wherein: The second video is modified by the additional user.

17. The computing device of claim 16, wherein: The second video includes text modified by the additional user.

18. The computing device of claim 11, wherein: The first subset of videos is ordered based on an ordering associated with videos in the pool of videos; and The first subset of videos is selected based on the ranking.

19. The computing device of claim 18, wherein: The ranking is based on the performance metric.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computing device, cause the computing device to: receiving, via a communication chat between a user of the computing device and another user of another computing device, a category selected by the user from a set of predetermined categories; selecting a first subset of videos from the pool of videos, a first video in the first subset of videos being associated with the category and having a calculated value of a performance metric; selecting a second subset of videos from the pool of videos, second videos in the second subset of videos being associated with the category and having an unknown value for the performance metric; generating a third video set based on a predetermined rule, the third video set comprising at least one video from the first video subset and at least one video from the second video subset; and The third set of videos is presented by the computing device via the communication chat.

Images