System and method for extracting temporal information from animated media content items using machine learning
The machine learning time analysis model extracts time information from animation media content projects, which solves the problem that existing systems cannot effectively extract time information from animation media content projects, and realizes the improvement of efficient media content classification and search results in the dynamic interface .
Patent Information
- Application Number
- CN202080005148.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-02-21
- Filing Date
- 2020-04-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-04-30
AI Technical Summary
Existing systems have difficulty providing classified animation media content within a dynamic interface without consuming resources or requiring manual intervention, and cannot effectively extract the time information conveyed in an animation media content project.
By using a machine learning time analysis model, time information is extracted from an animated media content project, which receives data describing media content projects, including multiple image frames for sequential display, and outputs time analysis data associated with these image frames. , used to identify and understand the semantic meaning and emotional content of animation media content.
It realizes providing more accurate and efficient media content classification in a dynamic interface, improves the relevance of search results, reduces the number of user searches, and saves computing resources and transmission bandwidth.
Smart Images

Figure CN113557521B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is a continuation of U.S. Provisional Patent Application Serial No. 62 / 979,624 filed on February 21, 2020, the disclosure of which is incorporated herein by reference in its entirety for all purposes. Technical Field
[0003] The present disclosure relates generally to computer vision and more particularly to systems and methods for extracting temporal information from animated media content items using machine learning. Background Art
[0004] Conventional methods for communicating between users of mobile devices may rely solely on SMS, messaging or "texting" through social networking applications. Internet or mobile device users can exchange messages through these various media. However, sometimes, users may wish to communicate via media content such as GIF (Graphics Interchange Format) or image files containing a set of static or animated images. Users can search for GIFs on the Internet, copy them through the native web browser of the operating system, and paste GIFs in various messaging applications. These conventional systems are not well suited for providing classified content in a dynamic interface without consuming resources or requiring manual intervention. In addition, such conventional systems cannot extract useful time information about animated media content items that is communicated to viewers of the media content items. Summary of the invention
[0005] Various aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.
[0006] According to one aspect of the present disclosure, a computer-implemented method may include: receiving data describing a media content item through a computing system including one or more computing devices, the media content item including multiple image frames for sequential display; inputting the data describing the media content item into a machine learning temporal analysis model through the computing system, the machine learning temporal analysis model being configured to receive the data describing the media content item and, in response to receiving the data describing the media content item, output temporal analysis data describing temporal information associated with sequentially viewing multiple image frames of the media content item; and receiving the temporal analysis data through the computing system and as an output of the machine learning temporal analysis model.
[0007] According to another aspect of the present disclosure, a computing system may include one or more processors and a machine learning temporal analysis model, the machine learning temporal analysis model being configured to receive data describing a media content item and, in response to receiving the data describing the media content item, output temporal analysis data describing temporal information associated with sequentially viewing multiple image frames of the media content item. The temporal information cannot be described by a single image frame of the multiple image frames. The computing system may include one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include: receiving data describing a media content item including multiple image frames for sequential display; inputting the data describing the media content item into the machine learning temporal analysis model; and receiving the temporal analysis data as an output of the machine learning temporal analysis model.
[0008] According to another aspect of the present disclosure, a computer-implemented method for training a machine learning temporal analysis model may include: receiving data describing a media content item by a computing system including one or more computing devices, the media content item including a plurality of image frames for sequential display; inputting the data describing the media content item into the machine learning temporal analysis model by the computing system, the machine learning temporal analysis model being configured to receive the data describing the media content item, and in response to receiving the data describing the media content item, outputting temporal analysis data describing temporal information associated with viewing a plurality of image frames of the media content item sequentially. The temporal information cannot be described by a single image frame of the plurality of image frames. The method may include: receiving the temporal analysis data by the computing system and as an output of the machine learning temporal analysis model; and adjusting one or more parameters of the machine learning temporal analysis model by the computing system based on a comparison of the temporal analysis data with baseline truth temporal analysis data.
[0009] Other aspects of the disclosure relate to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.
[0010] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] A detailed discussion of embodiments for one of ordinary skill in the art is set forth in the specification, which refers to the accompanying drawings, in which:
[0012] Figure 1ADepicted is a block diagram of an example computing system for extracting temporal information from an animated media content item using one or more machine learning models, according to an example embodiment of the present disclosure.
[0013] Figure 1B A block diagram of an example computing device for extracting temporal information from an animated media content item using one or more machine learning models is depicted according to an example embodiment of the present disclosure.
[0014] Figure 1C A block diagram of an example computing device for extracting temporal information from an animated media content item using one or more machine learning models is depicted according to an example embodiment of the present disclosure.
[0015] Figure 2 Depicted is a block diagram of an example machine learning temporal analysis model according to an example embodiment of the present disclosure.
[0016] Figure 3 Depicted is a flow chart of an example method for extracting temporal information from an animated media content item using one or more machine learning models according to an example embodiment of the present disclosure.
[0017] Figure 4 is a schematic diagram of an example animated media content item displaying a series of text strings according to an example embodiment of the present disclosure.
[0018] Figure 5A is a high-level block diagram depicting a media content management system according to aspects of the present disclosure;
[0019] Figure 5B is a high-level block diagram depicting a process for performing a search to implement animated input in a dynamic interface in accordance with aspects of the present disclosure;
[0020] Fig. 6A is a network diagram of a system for procuring, organizing, and retrieving expressive media content in a media content management system according to an embodiment according to aspects of the present disclosure, showing a block diagram of the media content management system;
[0021] Figure 6B is a high-level block diagram of a system for classifying obtained content to perform a search in a media content management system according to an embodiment in accordance with aspects of the present disclosure;
[0022] Figure 6C is a high-level block diagram of a system for arranging composite content items in a media content management system according to aspects of the present disclosure;
[0023] Fig. 7A is a high-level block diagram of a system for classifying obtained content in a media content management system in accordance with aspects of the present disclosure;
[0024] Figure 7B is a high-level block diagram of a system for performing a search to implement animated input in a dynamic interface in accordance with aspects of the present disclosure;
[0025] Fig. 8A -C is an example screenshot of a dynamic keyboard interface provided for interacting with content in a media content management system in accordance with aspects of the present disclosure; and
[0026] Fig. 9 An exemplary computing platform is shown, disposed in a device configured to obtain, organize, and / or retrieve expressive media content, in accordance with aspects of the present disclosure.
[0027] Reference numerals that are repeated across multiple drawings are intended to identify like features in the various embodiments. DETAILED DESCRIPTION
[0028] Overview
[0029] In general, the present disclosure relates to a system and method for extracting time information from an animated media content item using machine learning. When viewing certain animated media content items in an animated form, they can convey information to the viewer that may not be conveyed when analyzing any single frame of the animated media content item. A time analysis machine learning model can be trained to extract such time information. This information has various useful applications. For example, this time information can be used to identify "unsafe" (e.g., vulgar, offensive, etc.) media content items that may not appear unsafe when viewed or analyzed frame by frame. Extracting time information as described herein can help mark such items and / or remove such items from search results. In addition, extracting this time information can help improve the classification of items because media content items are better "understood" as users perceive them. As a result, better search results and suggestions can be provided to users.
[0030] More particularly, according to aspects of the present disclosure, a computer-implemented method for extracting temporal information may include receiving data describing a media content item, the media content item including a plurality of image frames for sequential display, such as a Graphics Interchange Format (GIF) file. For example, the data describing the media content item may define image components (such as edges or pixel attributes) of each image frame, and may be, for example, the GIF file itself. The method may include inputting the data describing the media content item into a machine learning temporal analysis model. The machine learning temporal analysis model may be configured to receive data describing the media content item, and in response, output temporal analysis data describing temporal information associated with a plurality of image frames for sequentially viewing the media content item. As described above, individual image frames in the plurality of image frames may not be able to convey or describe the temporal information. Therefore, analyzing or classifying the combined set of image frames may provide improvements relative to techniques applied to each frame in isolation.
[0031] For example, some media content items may include dynamic text captions, wherein, when the media content item is rendered in an animated form, the words or letters of the dynamic text caption are displayed sequentially. The animated media content item may include a first image frame containing a first text string (e.g., a first phrase, a word, a letter, etc.) and a second image frame containing a second text string (e.g., a second phrase, a word, a letter, etc.), and the second image frame is used to be displayed sequentially after the first image frame. When the media content item is rendered in an animated form, each word in the sentence or phrase may be displayed sequentially (e.g., flashing). The machine learning time analysis model may output time analysis data, which describes the semantic meaning of the complete dynamic text caption as perceived by the viewer of the animated media content item. In this example, the time analysis data may describe the semantic meaning of reading the first text string sequentially before the second text string. If the first text string and / or the second text string are read separately without evaluating the meaning when the text string is combined, the semantic meaning may be missing. Therefore, the time information output by the time analysis model can describe the semantic meaning of the entire dynamic text caption or media content item when read by the viewer.
[0032] As an example of such a dynamic caption, a first text string may include "good" and a second text string may include "Grief." When read or analyzed individually, "good" may convey happiness or joy, while "Grief" may convey sadness. A simple combination of analyzing each text string fails to capture the meaning of the dynamic caption as read by a viewer. However, when analyzed using a machine learning temporal analysis model, temporal information describing the meaning of the phrase "Good Grief" can be extracted that conveys an entirely different emotion, such as lighthearted frustration. Thus, the machine learning temporal analysis model can better extract meaning from the dynamic caption than analyzing the text strings individually as they appear in the animated media content item.
[0033] Additionally, portions of the dynamic text caption may sometimes change appearance, which contributes to the meaning of the dynamic text caption. For example, the appearance (e.g., position, color, size, font, etc.) of words and / or letters in the dynamic text caption may vary as the dynamic text caption is rendered in an animated form. These changes in appearance may convey meaning, such as emphasis and de-emphasis. For example, a particular word in the dynamic text caption may be emphasized by making the particular word larger, bolder, brighter in color, etc., compared to other words in the dynamic text caption. As is generally understood, a sentence may have a variety of different meanings depending on where the emphasis is placed in the sentence. The sentence "I NEVER SAID SHE STOLE MY MONEY" has been used as an example of how changing emphasis may change the meaning of a sentence. Depending on which word is emphasized, the sentence has seven different meanings. " I NEVER SAID SHE STOLE MY MONEY (I never said she stole my money) implies that someone else did steal the money. Never said she stole my money)" implies that someone accuses the speaker of stating "SHE STOLE MY MONE"; and so on. The temporal analysis model can be trained to extract temporal information describing the semantic meaning of the dynamic text caption based on the different appearances of various text strings of the dynamic text caption.
[0034] Similarly, (one or more) temporal analysis models can extract temporal information included in changing scenes of objects, people, etc. (e.g., with or without text). Multiple image frames may include a first image frame describing or depicting a first scene and a second image frame describing a second scene, the second image frame being used to be displayed sequentially after the first image frame. The temporal information can describe the semantic meaning of viewing the first scene sequentially before the second scene, which semantic meaning is not described or conveyed by viewing or analyzing the first scene or the second scene alone. In addition, it should be understood that the extracted temporal information can describe a combination of displaying a particular text description after a particular scene, or vice versa.
[0035] In some embodiments, the temporal analysis data may describe the emotional content of the media content item. Examples of different emotional content may include anxiety, fear, anger, happiness, sadness, jealousy, desire, interest, shock, boredom, surprise, relief, disgust, shame, pity, and any other human emotion. Even though each individual scene may not convey a particular emotion when viewed or analyzed frame by frame or scene by scene, a particular sequence of scenes may convey that particular emotion. The extracted temporal analysis data may describe that particular emotion.
[0036] In some embodiments, the temporal analysis model (e.g., a segmentation model thereof) may be configured to generate one or more media content item segments (e.g., corresponding to portions thereof) from the media content item. The temporal analysis data may be configured to generate intermediate temporal analysis data that describes temporal information associated with the segments of the media content item, respectively. The temporal analysis model may be configured to process the intermediate temporal analysis data to generate the temporal analysis data.
[0037] In some embodiments, the segmentation model may be configured to automatically segment the media content item based on caption appearances, changes, movements, etc. and / or based on scene changes in the media content item.
[0038] In addition, in some embodiments, the system (e.g., the time analysis model) can be configured to identify one or more segments of a media content item as corresponding to some or all of another media content item. The system can retrieve metadata and / or time analysis data of (one or more) segments from other media content items (e.g., their time analysis data and / or metadata). The acquired data (e.g., time analysis data and / or metadata) can be used to generate time analysis data for the media content item. For example, the acquired data can be analyzed, weighted, combined, interpolated, etc. to generate synthetic data describing the time content of the media content item itself. For example, the first metadata describing the content of the first segment of the media content item can be acquired, and the second metadata describing the content of the second segment of the media content item displayed after the first segment of the media content item 88 can be acquired. The time analysis data of the media content item can be generated based on an analysis of the effect of viewing the content described by the first metadata before the content described by the second metadata. Therefore, the segment time analysis data and / or segment metadata of the segment of the media content item can be used to generate the time analysis data of the media content item.
[0039] In some embodiments, media content items may include advertisements and / or sponsored content. Such media content items may be provided in response to a search query (e.g., within a dynamic keyboard interface) and / or as a suggestion to a user (e.g., as part of an auto-complete function for writing messages using a messaging application). Extracting time information as described herein may be particularly useful for sponsored media content. Improved classification and / or providing improved search results to search queries can motivate user interaction, thereby increasing the effectiveness of advertisements. For example, advertisements may be better classified by the semantic meaning of dynamic text descriptions and / or by emotional content. For example, advertisements depicting surprise (in a manner that is not easily determined using frame-by-frame analysis) may be provided in response to a search query associated with surprise. As a result, more relevant sponsored media content items may be provided in response to a search query, thereby increasing the interaction of users with the sponsored media content items.
[0040] The system and method of the present disclosure can provide many technical effects and benefits. For example, in response to a given search query, fewer and more accurate search results can be sent to a user computing device. As a result, less computing resources can be consumed or required to send and / or store search results (e.g., storage size, data transmission bandwidth, etc. on a user computing device and / or server computing device). In addition, users can more easily find (locate) appropriate or specific media content items, thereby reducing the number of searches that users need to perform. In addition, the analysis or classification of media content items can provide a more accurate description of the content of the media content items. For example, the data describing the media content items input into the machine learning time analysis model can describe or define the image components (such as edges or pixel attributes) of each image frame in a group of image frames, and can be, for example, the GIF file itself. The machine learning time analysis model can be trained to analyze the underlying image frame data to generate time analysis data output. The time analysis data output by the machine learning time analysis model can describe the subject displayed in the group of image frames (when displayed in animation form) in a more accurate manner than the data generated by the frame-by-frame analysis of the group of image frames. Therefore, the present application provides technical solutions (e.g., improved image and video processing and analysis) to technical problems (e.g., understanding, classification and / or categorization of media such as short videos such as GIFs).
[0041] For example, time information can describe, identify and / or be used to identify and / or detect the predetermined pattern that may not occur in the media content item when being analyzed frame by frame. Extracting time information as described herein can help to identify or detect such predetermined pattern that is usually undetectable when analyzing media content frame by frame, and helps to remove the media content item that is associated with and / or comprises the predetermined pattern from search results. As a result, search results can be provided more accurately and efficiently, thereby saving computing resources and transmission bandwidth.
[0042] More particularly, according to aspects of the present disclosure, a computer-implemented method for extracting temporal information may include receiving data describing a media content item, the media content item including a plurality of image frames for sequential display, such as a Graphics Interchange Format (GIF) file. The method may include inputting the data describing the media content item into a machine learning temporal analysis model. The machine learning temporal analysis model may be configured to receive the data describing the media content item and, in response, output temporal analysis data describing temporal information associated with sequentially viewing a plurality of image frames of the media content item.
[0043] In addition, the plurality of image frames may include a first image frame including a first digital image and a second image frame to be displayed sequentially after the first image frame, and the second image frame may include a second digital image. The temporal information described by the temporal analysis data may describe a pattern described by sequentially detecting the first digital image before the second digital image, which pattern is not described by detecting the first digital image or the second digital image alone.
[0044] Furthermore, the first digital image of the first image frame may have an appearance that is different from an appearance of the second digital image of the second image frame in at least one of color, coarseness, position, or shape. Furthermore, the temporal analysis data may describe a pattern associated with the first digital image having an appearance that is different from the appearance of the second digital image. The pattern associated with the first digital image having an appearance that is different from the appearance of the second digital image may be compared to a predetermined pattern, and if the pattern matches the predetermined pattern, the media content item associated with or including the predetermined pattern may be removed from the search results.
[0045] As an example, the systems and methods of the present disclosure may be included or otherwise used in the context of an application, a browser plug-in, or other environments. Thus, in some embodiments, the model of the present disclosure may be included in or otherwise stored and implemented by a user computing device such as a laptop computer, a tablet computer, or a smart phone. As yet another example, the model may be included in or otherwise stored and implemented by a server computing device that communicates with the user computing device according to a client-server relationship. For example, the model may be implemented by a server computing device as part of a web service (e.g., a web email service).
[0046] Referring now to the accompanying drawings, example embodiments of the present disclosure will be discussed in greater detail.
[0047] Example devices and systems
[0048] Figure 1A A block diagram of an example computing system 10 for extracting temporal information from an animated media content item using a machine learning model according to an example embodiment of the present disclosure is depicted. System 10 may include a user computing device 11, a server computing system 40, and a training computing system 60 communicatively coupled via a network 80.
[0049] The user computing device 11 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smart phone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.
[0050] The user computing device 11 includes one or more processors 12 and a memory 14. The one or more processors 12 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or a plurality of processors operatively connected. The memory 14 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 14 may store instructions 18 and data 16 that are executed by the processor 12 to cause the user computing device 11 to perform operations.
[0051] The user computing device 11 may store or include one or more temporal analysis models 20. For example, the temporal analysis model(s) 20 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other multi-layer nonlinear model. The neural network may include a recurrent neural network (e.g., a long short-term memory recurrent neural network), a feedforward neural network, or other forms of neural networks. Figure 2 Example temporal analysis model(s) 20 are discussed.
[0052] In some implementations, one or more temporal analysis models 20 may be received from a server computing system 40 via a network 80, stored in a user computing device memory 14, and used or otherwise implemented by one or more processors 12. In some implementations, a user computing device 11 may implement multiple parallel instances of a single temporal analysis model(s) 20 (e.g., to perform parallel analysis across multiple instances of a model 20).
[0053] Additionally or alternatively, one or more temporal analysis models 50 may be included in or otherwise stored and implemented by a server computing system 40 that communicates with the user computing device 11 according to a client-server relationship. For example, the temporal analysis model(s) 50 may be implemented by the server computing system 40 as part of a web service. Thus, one or more models 20 may be stored and implemented at the user computing device 11, and / or one or more models 50 may be stored and implemented at the server computing system 40.
[0054] The user computing device 11 may also include one or more user input components 22 that receive user input. For example, the user input component 22 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a conventional keyboard, or other means by which a user may input communications.
[0055] The server computing system 40 includes one or more processors 42 and memory 44. The one or more processors 42 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or multiple processors operatively connected. The memory 44 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 44 may store instructions 48 and data 46 that are executed by the processor 42 to cause the server computing system 40 to perform operations.
[0056] In some embodiments, server computing system 40 includes or is otherwise implemented by one or more server computing devices. Where server computing system 40 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.
[0057] As described above, the server computing system 40 may store or otherwise include one or more machine learning temporal analysis models 50. For example, the model 50 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep recurrent neural network) or other multi-layer nonlinear model. Figure 2 Discuss Example Model 50.
[0058] The server computing system 40 may train the models 20, 50 via interaction with a training computing system 60 that is communicatively coupled via a network 80. The training computing system 60 may be separate from the server computing system 40 or may be a part of the server computing system 40.
[0059] The training computing system 60 may include one or more processors 62 and a memory 64. The one or more processors 62 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or multiple processors operatively connected. The memory 64 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 64 may store instructions 68 and data 66 that are executed by the processor 62 to cause the training computing system 60 to perform operations. In some embodiments, the training computing system 60 includes or is otherwise implemented by one or more server computing devices.
[0060] The training computing system 60 may include a model trainer 70 that trains the machine learning model 50 stored at the server computing system 40 and / or the model 20 stored at the user computing device 11 using various training or learning techniques such as, for example, back propagation of errors. In some embodiments, performing back propagation of errors may include performing truncated backpropagation through time. The model trainer 70 may perform a variety of generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.
[0061] In some implementations, if the user has provided consent, training examples may be provided by the user computing device 11 (e.g., based on communications previously provided by the user of the user computing device 11). Thus, in such implementations, the model 20 provided to the user computing device 11 may be trained by the training computing system 60 on user-specific communication data received from the user computing device 11. In some cases, this process may be referred to as personalizing the model.
[0062] The model trainer 70 includes computer logic for providing the desired functionality. The model trainer 70 can be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some embodiments, the model trainer 70 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other embodiments, the model trainer 70 includes one or more sets of computer executable instructions stored in a tangible computer readable storage medium (such as a RAM hard disk or an optical or magnetic medium). The model trainer 70 can train (one or more) models 20, 50 based on the training data 72.
[0063] The network 80 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications through the network 180 may be carried via any type of wired and / or wireless connection using a variety of different communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).
[0064] Figure 1AAn example computing system that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some embodiments, the user computing device 11 may include a model trainer 70 and training data 72. In such embodiments, the model 20 may be trained and used locally at the user computing device 11. In some such embodiments, the user computing device 11 may implement the model trainer 70 to personalize the model 20 based on user-specific data.
[0065] Figure 1B A block diagram of an example computing device 82 is depicted that performs in accordance with an example embodiment of the present disclosure. The computing device 82 may be a user computing device or a server computing device.
[0066] The computing device 82 includes multiple applications (e.g., applications 1 to N). Each application contains its own machine learning library and (one or more) machine learning models. For example, each application can include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.
[0067] like Figure 1B As shown, each application can communicate with many other components of the computing device, such as, for example, one or more sensors, context managers, device state components, and / or additional components. In some embodiments, each application can communicate with each device component using an API (e.g., a public API). In some embodiments, the API used by each application is specific to the application.
[0068] Figure 1C A block diagram of an example computing device 84 performing in accordance with an example embodiment of the present disclosure is depicted. The computing device 84 may be a user computing device or a server computing device.
[0069] The computing device 84 includes a plurality of applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some embodiments, each application can communicate with the central intelligence layer (and the model(s) stored therein) using an API (e.g., a common API across all applications).
[0070] The central intelligence layer includes many machine learning models. For example, Figure 1CAs shown, a corresponding machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications can share a single machine learning model. For example, in some embodiments, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some embodiments, the central intelligence layer is included in the operating system of the computing device 84 or is otherwise implemented by it.
[0071] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for computing devices 84. Figure 1C As shown, the central device data layer can communicate with many other components of the computing device, such as, for example, one or more sensors, context managers, device state components, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).
[0072] Example Model Layout
[0073] Figure 2 A block diagram of an example temporal analysis model 86 according to an example embodiment of the present disclosure is depicted. In some implementations, the temporal analysis model 86 is trained to receive data describing a media content item 88 that includes a plurality of image frames for sequential display (e.g., a Graphics Interchange Format (GIF) file). The machine learning temporal analysis model 86 may be configured to receive data describing the media content item 88 and, in response, output temporal analysis data 90 describing temporal information associated with sequentially viewing the plurality of image frames of the media content item 88. Individual image frames of the media content item 88 may not be able to convey or describe the temporal information described by the temporal analysis data 90. The temporal analysis model 86 may also be configured to receive text recognition data 89 describing one or more text strings recognized in one or more image frames of the media content item 88. For example, as described below with reference to Fig. 6A and 7B As described above, the text recognition data 89 may be received from a natural language parser (NLP).
[0074] In some embodiments, temporal analysis model 86 (e.g., a segmentation model thereof) may be configured to generate one or more media content item segments from media content item 88. Temporal analysis data 90 may be configured to generate intermediate temporal analysis data describing temporal information respectively associated with the segments of media content item 88. Temporal analysis model 86 may be configured to process the intermediate temporal analysis data to generate temporal analysis data 90.
[0075] In some embodiments, the segmentation model may be configured to automatically segment the media content item 88 based on textual caption appearances, changes, movements, etc. and / or based on scene changes in the media content item 88 .
[0076] In addition, in some embodiments, the system (e.g., the temporal analysis model 86) can be configured to identify one or more segments of the media content item 88 as corresponding to some or all of another media content item. The system can obtain metadata and / or temporal analysis data of (one or more) segments from other media content items (e.g., their temporal analysis data and / or metadata). The acquired data (e.g., temporal analysis data and / or metadata) can be used to generate temporal analysis data for the media content item 88. For example, the acquired data can be analyzed, weighted, combined, interpolated, etc. to generate synthetic data describing the temporal content of the media content item 88 itself. For example, first metadata describing the content of a first segment of the media content item 88 can be obtained, and second metadata describing the content of a second segment of the media content item 88 displayed after the first segment of the media content item 88 can be obtained. The temporal analysis data of the media content item 88 can be generated based on an analysis of the effect of viewing the content described by the first metadata before the content described by the second metadata. Therefore, the segment temporal analysis data and / or segment metadata of the segments of the media content item 88 can be used to generate the temporal analysis data of the media content item 88.
[0077] Aspects of the present disclosure relate to training a temporal analysis model 86. One or more parameters of the temporal analysis model 86 may be adjusted based on a comparison of temporal analysis data 90 output by the temporal analysis model 86 to ground truth temporal analysis data 92. For example, a temporal analysis loss 94 may be calculated that describes a comparison between the temporal analysis data 90 and the ground truth temporal analysis data 92. Parameters of the temporal analysis model 86 may be iteratively adjusted based on the temporal analysis loss 94, for example using back propagation of errors.
[0078] Temporal analysis data 90 and / or ground truth temporal analysis data 92 may include or describe various information. For example, temporal analysis data 90 and / or ground truth temporal analysis data 92 may include or describe one or more content associations (e.g., as described below with reference to FIG. 5A to FIG. 5B as described above), the emotional content of the animated media content item, the meaning of the dynamic text description (e.g., including any emphasis), and the like.
[0079] Example Method
[0080] Figure 3Depicted is a flow chart of an example method performed according to an example embodiment of the present disclosure. Although for purposes of illustration and discussion, Figure 3 The steps are depicted to be performed in a particular order, but the method of the present disclosure is not limited to the particular order or arrangement shown. The various steps of the method 300 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.
[0081] At 302, a computing system may receive data describing a media content item that includes a plurality of image frames for sequential display, such as a Graphics Interchange Format (GIF) file. FIG. 5A to FIG. 7B As described, the computing system may receive (one or more) media content items from a media content store 106 , a media content source 124 , a third party application 202 , and / or a user device 102 .
[0082] At 304, the computing system may input data describing the media content item into a machine learning temporal analysis model, such as described above with reference to Figure 1A to Figure 2 The machine learning temporal analysis model may be configured to receive data describing a media content item and, in response, output temporal analysis data describing temporal information associated with sequentially viewing a plurality of image frames of the media content item. Individual image frames in the plurality of image frames may not be able to convey or describe the temporal information. The temporal analysis data may describe the semantic meaning, tone, and / or emotional content of the media content item and / or the dynamic text description included therein.
[0083] At 306, the computing system may receive temporal analysis data as an output of the machine learning temporal analysis model. The temporal analysis data may be used to identify "unsafe" (e.g., vulgar, offensive, etc.) media content items that may not appear unsafe when viewed or analyzed frame by frame, and / or may contribute to improved classification of media content items because the media content items may be better "understood" as the user perceives them. Thus, the temporal analysis data may be employed to improve the relevance of search results and / or suggestions for users that include media content items.
[0084] refer to Figure 4In some embodiments, a media content item may include a dynamic text description, wherein the words or letters of the dynamic text description are displayed sequentially when the media content item is rendered in an animated form. The animated media content item may include a first image frame 402 including a first text string 408 (e.g., a first phrase, word, letter, etc.) and a second image frame 404 including a second text string 410 (e.g., a second phrase, word, letter, etc.) to be displayed sequentially after the first image frame 402. The animated media content item may also optionally include a third image frame 406 including a third text string 412.
[0085] The dynamic text description can be displayed sequentially (e.g., as shown by arrows 414, 416) to connect the image frames 402, 404, 406. When the media content item is rendered in an animated form, the sentences can be displayed sequentially (e.g., flashing). The temporal analysis data can describe the semantic meaning of the complete dynamic text description as perceived by a viewer of the animated media content item. More specifically, the temporal analysis data can describe the semantic meaning of reading the first text string 408 sequentially before the second text string 410 (and the third text string 412, etc., if the third text string 412, etc. exists). This semantic meaning may be missing when reading the first text string 408 or the second text string 410 alone. Therefore, the time information output by the temporal analysis model can describe the semantic meaning of the entire dynamic text description or media content item when read by a viewer.
[0086] As an example of such a dynamic caption, a first text string may include "good" and a second text string may include "Grief". When read or analyzed individually, "good" may convey happiness or joy, while "Grief" may convey sadness. However, when analyzed as a dynamic caption, the phrase "Good Grief" conveys an entirely different emotion, such as lighthearted frustration.
[0087] Reference again Figure 4, certain parts of the dynamic text description may sometimes change appearance. For example, the appearance (e.g., position, color, size, font, etc.) of words and / or letters in the dynamic text description may be different as the dynamic text description is rendered in an animated form. These changes in appearance can convey meaning, such as emphasis. A particular word in the dynamic text description can be emphasized by making the particular word larger, bolder, brighter in color, etc. compared to other words in the dynamic text description. Such emphasis can affect the meaning of the dynamic text description. As is generally understood, the same sentence can have a variety of different meanings depending on which word(s) in the sentence are emphasized. For example, the sentence "I NEVER SAID SHE STOLEMY MONEY" has been used as an example of how emphasis can change the meaning of a sentence. Depending on which word is emphasized, the sentence has seven different meanings. " I NEVER SAID SHE STOLE MY MONEY (I never said she stole my money)" implies that someone else said it. "I NEVER SAID SHE STOLE MY MONEY (I Never said she stole my money)" implies that someone has accused the speaker of saying she stole his money; and so on. A temporal analysis model can be trained to extract temporal information describing the meaning of such different meanings based on the different appearances of the text strings of the dynamic text description.
[0088] As an example, the animated media content item may sequentially display three image frames 402, 404, 406. The first image frame 402 may include a first text string 408 "I NEVER SAID". The second image frame 404 may include a second text string 410 "SHE", and the third image frame 406 may include a third text string 412 "STOLE MYMONEY". One or more of the text strings (e.g., the second text string 410 in this example) may be emphasized by being displayed larger, bolder, more colorful, etc. than other words in the dynamic text caption.
[0089] The temporal analysis data can describe the meaning of the dynamic text description when read sequentially. Figure 4 In the depicted example, the temporal analysis data may include the semantic meaning imparted by the emphasis from the second text run 410 being different in appearance from the other text runs 408, 412. More specifically, in this example, the temporal analysis data may describe the semantic meaning suggesting that the speaker said someone else stole his money. Thus, the temporal analysis data may describe the semantic meaning of the dynamic caption when viewed sequentially by a reader, including the emphasis from the different visual characteristics of the text runs 408, 410, 412 of the dynamic caption.
[0090] Similarly, the temporal analysis model(s) may extract temporal information included in changing scenes (e.g., with or without text) of objects, people, etc. The plurality of image frames may include a first image frame describing or depicting a first scene and a second image frame describing a second scene to be displayed sequentially after the first image frame. The temporal information may describe the semantic meaning of viewing the first scene sequentially before the second scene, which semantic meaning is not described or conveyed by viewing or analyzing the first scene or the second scene alone.
[0091] In some embodiments, the temporal analysis data may describe the emotional content of the media content item. Examples of different emotional content may include anxiety, fear, anger, happiness, sadness, jealousy, desire, interest, shock, boredom, surprise, relief, disgust, shame, pity, and any and all other human emotions. Even though individual scenes may not convey a particular emotion when viewed or analyzed frame by frame or scene by scene, a particular sequence of scenes may convey that particular emotion.
[0092] In some embodiments, media content items may include advertisements and / or sponsored content. Such media content items may be provided in response to a search query (e.g., within a dynamic keyboard interface) and / or as a suggestion to a user (e.g., as part of an auto-complete function for writing messages using a messaging application). Extracting time information as described herein may be particularly useful for sponsored media content. Improved classification and / or providing improved search results to search queries can motivate user interaction, thereby increasing the effectiveness of advertising. For example, advertisements may be better classified by the semantic meaning of dynamic text descriptions and / or by emotional content. For example, advertisements depicting surprise may be provided in response to a search query associated with surprise. As a result, users are more likely to interact with sponsored media content items.
[0093] Example Configuration
[0094] Figure 5A is a high-level block diagram depicting a media content management system 100 according to some embodiments. The media content management system 100 may receive media content items 104 stored in a media content store 106 from a media content source 124 . Figure 5A The same reference numerals as in other figures are used to identify the same elements. A reference numeral followed by a letter, such as "102a", indicates that the text refers specifically to the element with that particular reference numeral. A reference numeral in the text that is not followed by a letter (such as "102") refers to any or all elements with that reference numeral in the figures (e.g., "102" in the text refers to reference numerals "102a" and / or "102b" in the figures). To simplify the description and to make it clear, Figure 5A Only two user devices 102 are shown. In one embodiment, an administrator can access the media content management system 100 through user devices 102 (eg, user devices 102a and 102b) through a separate login process.
[0095] As described above, media content items 104 may include various types of content, such as animated GIFs (a series of images), static images, audio-visual content items / videos, and composite content items, such as multiple animated GIFs and / or image content. Media content items 104 are received by the media content management system 100 and stored in the media content storage 106. Media content items 104 may have one or more attributes, such as content source, size, content brand (e.g., Paramount Pictures, NBC Universal, etc.), characters included in the content, text strings included in the content, etc. In one embodiment, the attributes may include metadata attributes.
[0096] In the media content storage 106, media content items 104 can be stored in association with a collection or grouping of media content items 104. In one embodiment, a collection can be generated by an administrator of the media content management system 100. In one embodiment, a collection can be automatically generated based on one or more attributes shared by the media content items 104 in the collection. In an embodiment, a content association or a unique identifier can be used to represent a collection in the media content management system 100. For example, a media content item 104 can be "content-associated" as part of a "#happy" collection in the media content management system 100. In one embodiment, a user or administrator can content-associate a media content item 104 as part of a "#happy" collection. In another embodiment, a content associator module 108 can automatically associate a media content item 104 with a media content item 104 using a content association stored in a content association storage 118. In this way, in the media content management system 100, content associations (such as "#happy") can be used to obtain and classify content. In the media content management system 100, each collection or file group can be tagged with a content association. In one embodiment, a particular file may be associated with one or more content associations.
[0097] The content associator module 108 may include one or more temporal analysis models 109, such as those described above with reference to Figure 2 The temporal analysis model described in temporal analysis model 86 of the present invention may be configured to analyze the media content items 104 and generate content associations for the media content items 104 based on temporal analysis data extracted from the media content items.
[0098] In one embodiment, a user of the media content management system 100 may add content to the media content management system 100 via the user device 102a. For example, the user may have installed an application extension 116 on the user device 102a, allowing the user to "save" content items 114 found by browsing a web page 112 using a browser 110 on the user device 102a. In one embodiment, by using the application extension 116 to save the content item 114, a URL (Uniform Resource Locator) may be stored in association with the content item 114 as a property of the content item. In one embodiment, the application extension 116 may include a downloadable application that enables a user to browse to a web page and collect media content items presented on the web page. As an example, a web page of a blog may publish particularly interesting content items that may or may not be available on the media content management system 100. Using the application extension 116, the user may browse to the web page 112, access a menu via the browser 110, and select an option to save one or more content items 114 that have been presented on the web page 112. In one embodiment, the application extension 116 is a mobile application that enables the mobile browser 110 to perform this function. In other embodiments, the application extension 116 may be a browser extension application or applet that may be downloaded via a browser 110 on a mobile device or desktop computer. In other embodiments, the application extension 116 may enable a user to upload content items 114 directly to the media content storage 106 in the media content management system 100.
[0099] In another embodiment, a copy of the content item 114 is stored in the media content storage 106 as part of the user operating the above-mentioned application extension 116. In other embodiments, a link or URL to the content item 114 is stored in the media content storage 106. In yet another embodiment, a copy of the content item 114 is stored on the user device 102a as part of a "saved" collection or a user-generated collection on the user device 102a. The user can log into his or her account on various user devices 102 so that collections, including user-generated collections such as "saved" collections, can be synchronized between the user devices 102.
[0100] In one embodiment, the content items 114 presented on the web page 112 or otherwise accessible by the web server can be obtained by the administrator of the media content management system 100 in other ways. For example, content owners (such as movie studios, television studios), brand owners and other content producers can cooperate with the administrator of the media content management system 100 so that the content of the license can be delivered and stored in the media content storage 106. In such a procurement process, the content owner can provide a media content item 104 with pre-filled attributes, as described above. The media content source 124 such as the content owner can include, for example, a content storage or database on a server maintained and operated by a third-party source or website. As part of the acquisition process, the content item 104 can be classified into one or more sets by storing the content item 104 in association with one or more content associations from the content association storage 118. In one embodiment, the content association can be automatically generated by the content associator module 108 based on the attributes of the content item 104. In another embodiment, the content association can be selected by one or more user interfaces or by an application programming interface (API). In other embodiments, the media content item 104 may be content associated by a user of the media content management system 100 after being stored in the media content store 106 through one or more user interfaces on the user device 102 .
[0101] like Figure 5AAs further shown, for example, a dynamic keyboard interface 122 may be provided on the user device 102b. The dynamic keyboard interface 122 may include a collection of media content items 104 and media content items 104. For example, the dynamic keyboard interface 122 may include a collection of media content items 104 that are content-associated as "#FOMO". "#FOMO" is an expression in Internet parlance, meaning "fear of missing out". Therefore, the media content items 104 included in the "#FOMO" collection may be about or include expressive statements about a particular expression "fear of missing out". In one embodiment, one or more expressive statements may be extracted from the media content items 104 and / or otherwise interpreted from them. For example, a curating user may content-associate a media content item 104 as "#FOMO" based on an image in the media content item 104 being associated with the expression "fear of missing out" (such as a blinking text of "FOMO" in an image, a textual description dialogue from a movie or television program indicating that the character in the image is lonely, has no friends, or is otherwise afraid of missing out on a cool event). Through the deriving process, expressive statements may be mapped to content associations in the media content management system 100. In one embodiment, these expressive statements may be related to the search intent of a user performing a search via an animated input in a dynamic interface.
[0102] like Figure 5A Shown, dynamic keyboard interface 122 can also include other animation keys (key) or the zone of the dynamic keyboard that realizes animation input.Except also including the "#FOMO" animation key of champagne bottle, also shown are two hands held in a handshake mode, baby crying, a pair of glasses, "#happy" content association and "#LOL" content association animation keys as example animation keys.Although not shown, the animation key can be included in the media content item 104 that is rendered as animation in the dynamic keyboard interface 122, which means that content can move in a key with continuous circulation.In one embodiment, media content item 104 can be pre-processed to be able to realize the animation input in the dynamic interface.
[0103] After selecting one of the animated keys in the dynamic keyboard interface 122, the user device 102b may communicate with the media content management system 100 via the search interface module 120. In one embodiment, the user's search history and / or the user's sharing history may be stored as personalized information in the personalized storage 150 for each user of the dynamic keyboard interface 122. Other personalized information about the user device 102 may be captured, such as location (via GPS and / or IP address), installed language keyboards, default language selection, phone information, contact information, installed messaging applications, etc. For example, the data included in the personalized storage 150 may be used by the search interface module 120 as one or more factors in determining the user's search intent. Figure 5B As further shown in FIG. 1 , the dynamic keyboard interface 122 can be rendered on the user device 102 b by a dynamic keyboard application 130 installed on the user device 102 b. The dynamic keyboard application 130 can install a dynamic keyboard user interface 132 that enables the dynamic keyboard interface 122 to be accessed throughout the user device 102 b as a third-party keyboard. In this way, a user who sends and receives messages using the messaging application 140 can access the dynamic keyboard interface 122 from within the messaging application 140.
[0104] Figure 5B is a high-level block diagram depicting the process of performing a search to implement animated input in a dynamic interface in an embodiment. Figure 5B As further shown in FIG. 1 , the media content items 104 are rendered in the dynamic keyboard interface 122 by the dynamic keyboard user interface 132 in communication with the search interface module 120. In one embodiment, a set of collections can be selected for display on the dynamic keyboard interface 122. Figure 5BAs shown, the dynamic keyboard interface 122 includes the collections of "#PLEASE", "#HAPPY", "#RUDE", and "#FACEPALM". Although the pound sign (#') is used in the examples included herein, the content association does not necessarily need to start with a pound sign. By selecting an animated key on the dynamic keyboard interface 122, the search interface module 120 can retrieve a collection of media content items 104 from the media content storage 106, and then the dynamic keyboard user interface 132 renders the collection of media content items 104 in the dynamic keyboard interface 122. In this way, a user performing a search will search the media content management system 100 by using a selected content association such as "#HAPPY". The retrieved collection of media content items 104 can be rendered within the dynamic keyboard interface 122. Since the "#HAPPY" collection can be updated and added in real time, different media content items 104 can be presented to the searching user when new items are added to the collection. As described above, the media content items 104 can be pre-processed to reduce the file size of the content, thereby enabling the media content items 104 to be quickly rendered on the dynamic keyboard interface 122.
[0105] Then, the user who is searching can select a media content item from the dynamic keyboard interface 122 by touching the dynamic keyboard user interface 132 or otherwise interacting with the dynamic keyboard user interface 132. Then, the selected media content item 144 can be sent or pasted into the messaging user interface 142 of the messaging application 140. In one embodiment, the selected media content item 144 is selected by clicking, tapping or touching the dynamic keyboard interface 122, and the selected media content item 144 is kept to "copy" the content so that it can be "pasted" into the messaging application 140 through the messaging user interface 142. In one embodiment, this copy and paste method can utilize the operating system of the user device 102 so that the selected media content item 144 is not permanently stored on the user device 102. In another embodiment, the user who is searching can search for media content through a search field on the dynamic keyboard interface 122, which will be further described herein. In this way, the media content item 104 can be shared through any messaging platform available on the user device. As described above, personalization information may also be captured in personalization storage 150, such as by search interface module 120. In at least some embodiments, dynamic keyboard interface 122 may be implemented as a GIF keyboard produced by RIFFSY, Inc. of San Francisco, California.
[0106] Fig. 6A1 is a network diagram of a system for classifying obtained content to perform a search in a media content management system according to an embodiment, showing a block diagram of a media content management system. The system environment includes one or more user devices 102, media content sources 124, third-party applications 202, media content management system 100, and network 204. In alternative configurations, different and / or additional modules may be included in the system.
[0107] The user device 102 may include one or more computing devices that may receive user input and may send and receive data via the network 204. In another embodiment, the user device 102 may be a device with computer functionality, such as a personal digital assistant (PDA), a mobile phone, a smart phone, a wearable device, etc. The user device 102 is configured to communicate via the network 204. The user device 102 may execute an application, such as a browser application that allows a user of the user device 102 to interact with the media content management system 100. In another embodiment, the user device 102 interacts with the media content management system 100 through an application programming interface (API) running on a native operating system of the user device 102.
[0108] In one embodiment, the network 204 uses standard communication technologies and / or protocols. Thus, the network 204 may include links using technologies such as Ethernet, 802.11, Worldwide Interoperability for Microwave Access (WiMAX), 3G, 4G, CDMA, Digital Subscriber Line (DSL), etc. Similarly, the networking protocols used on the network 204 may include Multi-Protocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), and File Transfer Protocol (FTP). Data exchanged through the network 204 may be represented using technologies and / or formats including Hypertext Markup Language (HTML) and Extensible Markup Language (XML). In addition, all or some of the links may be encrypted using conventional encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), and Internet Protocol Security (IPsec).
[0109] Fig. 6AA block diagram of a media content management system 100 is included. The media content management system 100 includes a media content storage 106, a content association storage 118, a personalized storage 150, a search interface module 120, a content associator module 108, a dynamic keyboard interface module 208, a web server 210, a dynamic keyboard rendering module 212, a content association management module 214, a sentiment analysis module 220, an image analyzer module 222, a motion analyzer 224, a natural language processing (NLP) parser 218, a heuristic engine 216, and a search router rule engine 206. In other embodiments, the media content management system 100 may include additional, fewer, or different modules for various applications. Conventional components such as network interfaces, security functions, load balancers, failover servers, management and network operation consoles, etc. are not shown to avoid obscuring the details of the system.
[0110] The web server 210 links the media content management system 100 to one or more user devices 102 via the network 204; the web server 210 provides web pages and other web-related content, such as Java, Flash, XML, etc. The web server 210 may provide functionality to receive and route messages between the media content management system 100 and the user devices 102, such as instant messages, queued messages (e.g., emails), text and SMS (Short Message Service) messages, or messages sent using any other suitable messaging technology. A user may send a request to the web server 210 to upload information, such as images or media content stored in the media content storage 106. In addition, the web server 210 may provide API functionality to send data directly to the native user device operating system.
[0111] The content associator module 108 can automatically generate one or more content associations for the media content items 104 in the media content management system 100 based on the properties of the media content items 104. For example, the content associator module 108 can use machine learning techniques to determine the relationship between the media content items 104 and the content associations stored in the content association storage 118.
[0112] The content associator module 108 may include one or more temporal analysis models 109, such as those described above with reference to Figure 2 The content associator module 108 may receive data describing text recognized in one or more image frames of a media content item from the NLP parser 218 (e.g., Figure 2 The text recognition data 89) can be input into the time analysis model 86.
[0113] The content associator module 108 may be configured to analyze the media content items 104 and generate content associations for the media content items 104 based on temporal analysis data extracted from the media content items.
[0114] In one embodiment, content associator module 108 can identify one or more content sources, such as movie studios, movies, TV studios, TV shows, actors, genres, etc. In another embodiment, content associator module 108 can automatically generate content associations for media content items 104 based on analysis of image frames in media content items 104. In another embodiment, content associator module 108 can use one or more computer vision techniques and other image processing methods. For example, various third-party applications 202 can be used to analyze image frames in media content items 104, and automatically generate one or more content associations to be associated with content items. In one embodiment, content associator module 108 can utilize one or more third-party applications 202, NLP parser 218, sentiment analysis module 220, image analyzer 222, motion analyzer 224 and heuristic engine 216 to analyze and parse the text included in media content items 104 and analyze the motion image frames of media content items 104 to automatically generate content associations and / or automatically select content associations stored in content association storage 118. In another embodiment, the NLP parser 218 can be combined with the sentiment analysis module 220 and can be relied upon to analyze images and / or audiovisual content to determine the sentiment of the media content item 104. For example, the image analyzer 222 and the motion analyzer 224 can be used to detect and / or classify image sequences depicting smiling faces. The heuristic engine 216 can include a rule that automatically associates a media content item 104 having an image sequence that has been analyzed to detect a smile with a "#happy" content association from the content association storage 118 when the media content item 104 is stored in the media content storage 106 in the media content management system 100. Alternatively, or in addition to this analysis, the NLP parser 218 can parse a text string included in the image and determine a match with the word "AWESOME". In addition, the NLP parser 218 can interpret a smile as representing a positive sentiment. The sentiment analysis module 220 may indicate that the word "AWESOME" is associated with strong positive sentiment, and the heuristic engine 216 may include rules that automatically associate a "#happy" content association (and / or other positive content associations) with media content items 104 having strong positive sentiment.
[0115] In an embodiment, the search interface module 120 may manage search requests and / or search queries for media content items 104 in the media content management system 100 received from the user device 102. In one embodiment, the search query may be received at the search interface module 120 and processed by the search router rule engine 206. In another embodiment, as a result of selecting an animation key or a text search, the search interface module 120 may receive a request for a collection based on content associations such as "#HAPPY", "#RUDE", "#FOMO", etc. from the user device 102. In an embodiment, the search interface module 120 may communicate the search query to the search router rule engine 206 to process the request.
[0116] The content association management module 214 can manage one or more content associations associated with each media content item 104 in the media content management system 100. The content associations can be associated with the media content items 104 through various interfaces, such as user interfaces and application programming interfaces (APIs), through the content association management module 214. The API can be used to receive, access, and store data from media content sources 124, third-party applications 202 (and / or websites), and user devices 102. In one embodiment, the content association management module 214 can manage how the content associations are associated with the media content items 104 through various acquisition methods.
[0117] The dynamic keyboard interface module 208 can manage the interface communication between the media content management system 100 and the user device 102. For example, Figure 5A and 5BAs shown, the dynamic keyboard interface 122 may include a menu selection element that enables a user who is searching to view trending media content on the media content management system 100. "Trending" media content may include content that is frequently viewed and / or frequently shared by users of the media content management system 100. For example, the dynamic keyboard interface module 208 may receive a request for trending media content and retrieve the media content item 104 with the highest number of shares in the past hour from the media content storage 106. In one embodiment, the dynamic keyboard interface module 208 may then provide the retrieved trending media content items to the dynamic keyboard interface 122 through the dynamic keyboard application 130 via the dynamic keyboard presentation module 212. For example, the dynamic keyboard presentation module 212 may determine how to present the media content items and in what order to present the media content items. In one embodiment, if no media content item 104 satisfies the search query or request from the user device, the dynamic keyboard interface module 208 may deliver other media content items 104 that are popular or shared in conjunction or in collaboration with the search interface module 120 and the search router rule engine 206. In one embodiment, the dynamic keyboard interface module 208 can select content items from the third-party application 202 (or website) to include in the animated keys of the dynamic keyboard interface 122 or in search results.
[0118] Heuristic engine 216 can include one or more heuristic rules to determine one or more results. For example, content associator module 108 can use heuristic engine 216 to determine the ranking of candidate content association of media content project 104 based on the attribute of media content project 104. Certain attributes can have various heuristic rules associated with them, such as visual actions (for example, detected smile can be associated with "#HAPPY" content association), visual characteristics (for example, flashing text can indicate the importance of text string, or pound sign can indicate specific content association), content source, characters included in media content project and other attributes. In one embodiment, administrator can generate various heuristic rules to automatically generate content association of content project based on attribute. In another embodiment, heuristic rules can also use parameter ranges of various attributes. For example, thirty selections of media content project 104 by a specific user for sharing can be used in heuristic rules to present the same media content project in response to the search query from the specific user when there are a few search results. For example, the scope here can be defined as the threshold number of times of sharing.
[0119] The sentiment analysis module 220 can provide analysis of various texts received by the media content management system 100 to determine whether the text exhibits positive, negative, or neutral meanings. This information can be used by various modules to efficiently convert search queries to extract the expressive intent of the user performing the search. For example, a term dictionary in multiple languages can be used to determine whether the text can be determined to have positive, negative, or neutral meanings. In one embodiment, the sentiment analysis module 220 can use various third-party applications 202 to perform the analysis. For example, using the sentiment analysis module 220, the search router rule engine 206 can provide one or more sets of media content items 104 based on the meaning of the search query.
[0120] Figure 6B is a high-level block diagram of a system for classifying obtained content to perform searches in a media content management system according to an embodiment. In one embodiment, the content association management module 214 may include a metadata analyzer module 240, a user interface module 242, a content association selection module 244, and an association correlation module 246.
[0121] When a media content item 104 having one or more attributes is received from a media content source 124 in the media content management system 100, the metadata analyzer module 240 can generate one or more content associations based on the attributes of the media content item 104. For example, the media content item 104 can be automatically content-associated in a collection of specific movies based on the movie metadata attributes associated with the media content item 104 from the specific movie. In one embodiment, an administrator of the media content source 124 can associate one or more metadata attributes to the media content item 104. The metadata attributes can be stored in the source files of the media content items 104 in various ways, such as header content associations within the source files, and other files associated with the source files, such as XML files that describe content items obtained in batches by the media content management system 100.
[0122] In one embodiment, metadata analyzer module 240 may parse metadata associated with media content item 104 and automatically generate content associations and / or select content associations from content association storage 118 based on one or more rules. Figure 6B As shown, the content association storage 118 can store the association-attribute relationship 250 so that the attribute has been associated with the content association. In this way, the metadata analyzer module 240 can automatically assign content associations to the media content items 104 based on the association-attribute relationship 250 stored in the content association storage 118.
[0123] Other metadata attributes that may be analyzed by the metadata analyzer module 240 include the Internet Protocol (IP) address of the mobile device or user device used by the user conducting the search or the administrative user. The IP address may provide an indication of the user's geographic location, including the country of origin. Alternatively, the global positioning system (GPS) of the mobile device may include the user's current geographic location. As a result, different collections or content associations may be presented to the user based on the primary language spoken at the user's geographic location. In another embodiment, another metadata attribute that may be analyzed by the metadata analyzer module 240 includes one or more languages selected by the user conducting the viewing. In this way, language preferences may help inform search intent, administrative intent, or both search intent and administrative intent. For example, a word in French may have a completely different meaning in Indonesian. As a result, language and country of origin may be metadata attributes that may be determined by the metadata analyzer module 240.
[0124] The user interface module 242 may provide one or more user interfaces for a user device 102, such as a computer or mobile device, to select one or more content associations for an acquired media content item 104. For example, an administrative user may be given the ability to assign one or more content associations from the content association storage 118 to a media content item 104. In this manner, the content association management module 214 enables manual selection of content associations to categorize an acquired media content item 104.
[0125] According to an embodiment, content association selection module 244 may provide one or more content associations from content association storage 118 in one or more user interfaces provided by user interface module 242. In one embodiment, content association selection module 244 may present predicted content associations based on content association-attribute relationships 250 stored in content association storage 118 for selection and / or confirmation by an administrative user operating user device 102. For example, media content item 104 may have a genre attribute of comedy based on pre-populated information from media content source 124. In one embodiment, media content item 104 may have been assigned a "#HAPPY" content association by metadata analyzer module 240 because the "comedy" attribute may be associated with a "#HAPPY" content association. Content association selection module 244 may present the "#HAPPY" content association along with other related content associations in a user interface provided by user interface module 242 for an administrative user to assign or revoke a content association associated with the associated content item 104. In one embodiment, association-attribute relationships 250 stored in content association storage 118 may include content associations that are related to other content associations. For example, the "#HAPPY" content association may be related to the "LOL" and "LMAO" content associations because both LOL and LMAO include a "laugh" interpretation. As a result, in one embodiment, other content associations may be presented for selection by the administrative user.
[0126] The content association selection module 244 may include one or more temporal analysis models 245, for example, as described above with reference to Figure 2 The temporal analysis model described in temporal analysis model 86 of the embodiment of the present invention may be used to select content associations of the media content items 104. The content association selection module 244 may be configured to analyze the media content items 104 and generate content associations for the media content items 104 based on temporal analysis data extracted from the media content items 104.
[0127] As part of the acquisition process, the media content items can be pre-processed 252 before being stored in the media content storage 106. This enables the media content items 104 to be quickly retrieved and seamlessly rendered in the dynamic keyboard interface 122 on the user device 102. The pre-processing 252 of the media content items can include reducing pixel count, modifying resolution clarity and other file size reduction techniques. In one embodiment, the dynamic keyboard rendering module 212 can be used to perform this pre-processing 252 of the media content items. Advantageously, the pre-processing 252 of the media content items enables the dynamic keyboard interface 122 presented to the user on the user device 102b to render at least two renderings of at least two media content items in an animated form, and they are displayed in the dynamic keyboard interface 122 simultaneously.
[0128] The association module 246 may associate a content association with a media content item 104 in the media content store 106. The content association may be automatically associated with the content item by the metadata analyzer module 240 (or other modules in the media content management system 100), or the content association may be associated as a result of a selection of the content association received through a user interface provided by the user interface module 242. Figure 6B As shown, item association relationships 254 are stored in the media content store 106. Each content item may have a content identifier, and each content association may have a content association identifier, so that the item-association relationships 254 may be stored in the media content store 106. Figure 6B As shown, a content item (“item”) may be associated with one or more associations (“associations”), and item-association relationships 254 are stored in the media content store 106 , for example.
[0129] Figure 6C It is a high-level block diagram of a system for arranging a composite content item in a media content management system according to an embodiment. A composer interface 264 can be provided on a user device 102, and this composer interface 264 enables the user who views to search for media content items 104 and selects two or more content items to generate a composite content item. As shown, two content items have been selected in the composer interface 264 to create a composite content item 266 with a combined attribute of the content items of two selections. For example, the user who views can search for "No" through a search interface, which is described in more detail later. Several content items 104 that meet the search term "No" can be retrieved. The content item of the first selection may have been associated with the content association of "No" and "Chandler", and the content item of the second selection may have been associated with the content association of "No" and "Taylor". As a result, a composite content item 266 may include content associations "No", "Chandler" and "Taylor". Composite content item 266 may be received by organizer interface module 262 and stored by composite item module 260 as media content item 104 in media content storage 106. Figure 6C As further shown, in addition to the arranger interface module 262, the composite project module 260 may also operate in conjunction with or include the metadata analyzer module 240, the content association selection module 244, and the association correlation module 246 that operate similarly as described above.
[0130] The content association selection module 244 may include one or more time analysis models 245, such as those described above with reference to Figure 2The temporal analysis model described in temporal analysis model 86 of the embodiment of the present invention may be used to select content associations of the media content items 104. The content association selection module 244 may be configured to analyze the media content items 104 and generate content associations for the media content items 104 based on temporal analysis data extracted from the media content items 104.
[0131] In at least some embodiments, the composite content item 266 may be associated with an expressive statement that conveys a different meaning than the individual content items included in the composite content item 266. Returning to the example above, the first content item 104 in which the character "Chandler" expresses the statement "No" may convey a particular meaning to most users of the media content management system 100. An administrative user of the media content management system 100 may associate other content associations with the particular content item 104, such as "#cool" and "FRIENDS". The second content item 104 depicting the celebrity TAYLOR LAUTNER may evoke a separate and different meaning than the first content item 104 depicting the character "CHANDLER" from the television show FRIENDS. In addition to the shared content association of "No", the second content item 104 may also be automatically or manually associated with content associations such as "cool" and / or "famous". As a result, the combination of the two media content items presents information that is different from each media content item presented separately. In one embodiment, the expressive statement presented by composite content item 266 may be a simple conglomeration of content associations associated with individual content items included in composite content item 266. In another embodiment, an expressive statement different from the content associations included in the individual content items may be extracted or otherwise interpreted from composite content item 266. As described herein, when the content associations are stored via the association associated with composite content item 266, the expressive statement will be used to correlate the intent of the user conducting the search with related content items.
[0132] Fig. 7A 1 is a high-level block diagram of a system for classifying obtained content in a media content management system according to some examples. The content associator module 108 may include a content association algorithm 406 for automatically selecting content associations 402 for the media content item 104. The content associator module 108 may also include a content association selector 408 for selecting the content association 402 from the content association storage 118. The content associator module 108 may operate in conjunction with or include the image analyzer 222, the motion analyzer 224, and the heuristic engine 216 to assist in automatically selecting the content association 402 for the media content item 104.
[0133] The content associator module 108 may include one or more temporal analysis models 245, such as those described above with reference to Figure 2 The temporal analysis model described in temporal analysis model 86 of the embodiment of the present invention can be used to analyze the media content items 104 and generate content associations for the media content items 104 based on the temporal analysis data extracted from the media content items 104.
[0134] The image analyzer 222 may include computer vision techniques for identifying facial features, such as faces, eyes, smiling mouths, frowning brows, etc. The image analyzer 222 may also include other computer vision techniques and / or pattern recognition algorithms to create a baseline training set for identifying these facial features. Similarly, the motion analyzer 224 may include computer vision techniques and / or pattern recognition algorithms, as well as machine learning and Bayesian inference techniques to identify crying, laughing, falling, and other actions that can be modeled in a similar manner. The motion analyzer 224 may also include eye tracking capabilities to identify the position of eyes within a set of images or animated images. The eye tracking capabilities of the motion analyzer 224 may be used in conjunction with one or more other modules in the media content management system 100, for example, to generate new media content items 104, such as rendering a pair of sunglasses onto a set of animated images, above the eyes detected within these images. Other modules may be used to add text to the media content items 104, such as the phrase "deal with it" to create and / or generate new media content items 104. As previously described, the heuristic engine 216 may draw conclusions based on the received data using various rules. For example, Fig. 7A As shown, media content item 104 may include, for example, a GIF of a baby crying. Image analyzer 222 may analyze frames of the GIF of media content item 104 and determine facial features such as eyes squinting, an open mouth in a frown-like position, and raised eyebrows. Motion analyzer 224 may identify that media content item 104 includes a baby crying based on a baseline model of a baby crying and other machine learning techniques.
[0135] As a result, the content associator module 108 may select one or more content associations from the content association store 118 via the content association selector 408. The content association algorithm 406 may include one or more heuristic rules from the heuristic engine 216 to automatically generate content associations for the media content item 104. In this example, the "#sad" content association 402 has been selected for the media content item 104. As previously described, the content association may be associated with other content associations, such as the crying content association may be associated with the "#sad" content association 402. In this manner, the media content item 104 of the baby crying may be included in the "#sad" collection 404 and stored in the media content store 106 based on the automatically generated content associations.
[0136] Figure 7B 6 is a high-level block diagram of a system for performing searches to implement animated input in a dynamic interface according to some examples. The search router rule engine 206 may include a query analyzer 602, an intent extractor 604, an intent matcher 606, and a machine learning module 608. In one embodiment, the query analyzer 602 may decompose the received text and / or picture into overlapping windows. For example, a user performing a search may enter the search term "happy birthday" as a query. The query analyzer 602 may decompose the query into words and overlapping partial words, such as "ha", "happy", "birth", "birthday", "happy birth", and "happy birthday". In one embodiment, the query analyzer 602 may provide the words and partial words to the search interface module 120 for searching for content associations of associated media content items in the media content storage 106 based on the words and partial words.
[0137] In another embodiment, the query analyzer 602 may provide words and partial words to the intent extractor 604. For example, the intent extractor 604 may have previously mapped or extracted intent from the query "happy birthday" to include the intent of celebrating a birthday. Therefore, the word "happy birthday" may be specifically mapped only to content items with birthday elements, such as cakes, candles, the text string "happy birthday", parties, people blowing candles, etc. The intent extractor 604 may also provide words and partial words to the natural language processing (NLP) parser 218 to derive meaning and / or intent from the search terms. In an embodiment where the search terms cannot be identified, the NLP parser 218 may be particularly useful. For example, if the search term is "happy dia de los muertos", and the word "dia de los muertos" (Spanish for "Day of the Dead") is not included in the dictionary or corpus of learned words, the intent extractor 604 may extract the intent of the user who is searching to celebrate something happy based on the word "happy" being included in the search query. On the other hand, if "muertos" is included in the dictionary or as a text string included as a metadata attribute of the content item, then the content item associated with the "happy" and "muertos" content association may be presented by relying on the NLP parser 218 .
[0138] In one embodiment, an intent matcher 606 may be used in the search router rule engine 206 to match the intent of the user conducting the search with one or more content associations in the content association storage 118. Returning to the previous example, the word "happy" included in the search query "happy dia de los muertos" may cause the search query to be matched by the intent matcher 606 to the "#happy" content association for further querying. In one embodiment, the word "muertos" may be matched to the "dead" content association and the "Halloween" content association. Because "dia de losmuertos" is not directly related to Halloween, but is actually a Mexican holiday that occurs on November 1, certain content items may not be presented. In one embodiment, the intent matcher 606 may adjust the matching between the search phrase and the content association. In one embodiment, these matches may be stored in the content association storage 118.
[0139] In another embodiment, the intent matcher 606 can be combined with the machine learning module 608 to analyze user feedback, such as selecting content items with both the "Halloween" attribute and the "skull" attribute when these items are presented in the search results in response to the "happy dia de los muertos" search query. As a result, the intent matcher 606 can generate new matches between the search phrase "happy dia de los muertos" and content items with both "Halloween" and "skull" content associations. In one embodiment, the intent matcher 606 can determine the likelihood score of the intent match for each match based on probabilistic methods and / or machine learning. The score can be stored in the content association storage 118 for each intent match. These scores can be further based on the statistical inference algorithms provided by the NLP parser 218 and the machine learning module 608.
[0140] The machine learning module 608 may use various machine learning methods, such as supervised and unsupervised learning methods, Bayesian knowledge bases, Bayesian networks, nearest neighbors, random walks, and other methods, to determine various results based on the received training data and the received user feedback (based on whether the viewing user selected / shared the content item presented in the search result group). For example, sometimes the random content item is presented together with the content item having a specific attribute (such as the "#happy" content association). At other times, the same content item may be randomly presented in the search results for different content associations (such as "dog"). The randomly presented content item may not be associated with the "#happy" content association or the "dog" content association, but the user who searches and / or views may frequently select and share the randomly presented content item. As a result, the machine learning module 608 may determine that the randomly presented content item is selected 80% of the time overall, 70% of the time when presented with content associated with "#happy", and 60% of the time when presented with content associated with "dog". The machine learning module 608 can be used to further automate the process and create heuristic rules to automatically present the content item when the search query includes the words "#happy" and "dog" and when the search query contains one of these words. In one embodiment, the machine learning module 608 can associate or relate content associations to the content item based on the content item being selected in search results having a common attribute within a threshold percentage of the time, such as 50%. According to at least one embodiment, associations such as these may also require administrator approval through a user interface.
[0141] The machine learning module 608 may include one or more temporal analysis models 245, such as those described above with reference to Figure 2 The machine learning module 608 may be configured to analyze the media content item 104 and generate content associations for the media content item 104 based on temporal analysis data extracted from the media content item 104.
[0142] The search router rule engine 206 may also include rules for processing search queries to optimize processing time and include search results even when there is no direct match in the media content management system 100. For example, the search router rule engine 206 may operate in conjunction with the sentiment analysis module 220, the image analyzer 222, and / or the motion analyzer 224 to analyze content items in the media content storage 106 that do not have associated attributes. The sentiment analysis module 220 may be used to process words, partial words, and search queries to determine whether the intent includes positive, negative, or neutral meanings. The image analyzer 222 may similarly be used to process received images received as search queries to extract the intent of the user conducting the search. For example, if the image is a photo captured by a mobile device that is sent directly as a query, the image analyzer 222 may analyze the photo to detect visual characteristics, such as facial expressions and activities occurring in the photo. In addition, the motion analyzer 224 may be used to detect patterns of actions, behaviors, and movements, such as laughing, crying, falling, shaking hands, punching, beating chests, rolling eyes, twirling hair, etc. Rules may be included in the search router rules engine 206 to associate the identified behaviors, actions, activities, and / or facial expressions with one or more expressive statements stored as content associations in the content association store 118. In one embodiment, these rules may be heuristic rules generated by the heuristic engine 216.
[0143] Fig. 9 An exemplary computing platform disposed in a device configured to classify obtained content to perform a search in a media content management system 100 is shown according to various embodiments. In some examples, the computing platform 1000 can be used to implement a computer program, application, method, process, algorithm, or other software to perform the above-described techniques.
[0144] In some cases, the computing platform may be provided in a wearable device or implement mobile computing device 1090b or any other device, such as computing device 1090a.
[0145] The computing platform 1000 includes a bus 1004 or other communication mechanism for transmitting information, which interconnects subsystems and devices such as a processor 1006, a system memory 1010 (e.g., RAM, etc.), a storage device 1008 (e.g., ROM, etc.), a communication interface 1012 (e.g., Ethernet or wireless controller, Bluetooth controller, etc.) to facilitate communication via ports on a communication link 1014, thereby, for example, communicating with computing devices including mobile computing and / or communication devices having a processor. The processor 1006 can be implemented with one or more central processing units ("CPUs") or one or more virtual processors and any combination of CPUs and virtual processors. The computing platform 1000 exchanges data representing input and output via input and output devices 1002, which include but are not limited to keyboards, mice, audio inputs (e.g., voice to text devices), user interfaces, displays, monitors, cursors, touch-sensitive displays, LCD or LED displays, and other I / O related devices.
[0146] According to some examples, the computing platform 1000 performs specific operations by executing one or more sequences of one or more instructions stored in the system memory 1010 by the processor 1006, and the computing platform 1000 can be implemented in a client-server arrangement, a peer-to-peer arrangement, or implemented as any mobile computing device, including smart phones, etc. Such instructions or data can be read into the system memory 1010 from another computer-readable medium such as the storage device 1008. In some examples, hard-wired circuits can be used instead of software instructions or in combination with software instructions. Instructions can be embedded in software or firmware. The term "computer-readable medium" refers to any tangible medium that participates in providing instructions to the processor 1006 for execution. Such media can take many forms, including but not limited to non-volatile media and volatile media. Non-volatile media include, for example, optical disks or magnetic disks, etc. Volatile media include dynamic memory, such as system memory 1010.
[0147] Common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with hole patterns, RAM, PROMs, EPROMs, flash EPROMs (FLASH-EPROMs), any other memory chips or boxes, or any other media from which a computer can read. Transmission media may also be used to send or receive instructions. The term "transmission media" may include any tangible or intangible medium capable of storing, encoding, or carrying instructions for machine execution, and may include digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Transmission media include coaxial cables, copper wires, and optical fibers, including wires containing bus 1004 for sending computer data signals.
[0148] In some examples, execution of the sequence of instructions may be performed by computing platform 1000. According to some examples, computing platform 1000 may be connected to a network through communication link 1014 (e.g., a wired network such as a LAN, PSTN, or any wireless network, including WiFi, Zig-Bee, etc.) to any other processor to execute instruction sequences in coordination (or asynchronously) with each other. Computing platform 1000 can send and receive messages, data, and instructions, including program code (e.g., application code), through communication link 1014 and communication interface 1012. The received program code can be executed by processor 1006 when it is received and / or stored in memory 1010 or other non-volatile storage device for later execution.
[0149] In the example shown, the system memory 1010 may include various modules that include executable instructions for implementing the functionality described herein. The system memory 1010 may include an operating system ("O / S") 1030, as well as applications 1032 and / or logic modules 1050. In the example shown, the system memory 1010 includes a content associator module 108 that includes a content association ("ass'n") selector module 408 and a content association ("CA") algorithm module 1040. The content associator module 108 may include one or more temporal analysis models 245, such as those described above with reference to Figure 2 The temporal analysis model described in temporal analysis model 86 of the embodiment of the present invention can be used to analyze the media content items 104 and generate content associations for the media content items 104 based on the temporal analysis data extracted from the media content items 104.
[0150] The system memory 1010 may also include an image analyzer 222, a motion analyzer 224, a heuristic engine 216, a search interface module 120, a dynamic keyboard interface module 208, a dynamic keyboard rendering module 212, a sentiment analysis module 220, a natural language processing (NLP) parser 218, a search router rule engine 206 (including a query analyzer 602, an intent extractor 604, an intent matcher 606, and a machine learning (ML) module 608), a content association ("ass'n") management ("mgmt.") module 214 (including a metadata analyzer module 240, a user interface module 242, a content association selection module 244, and an association ("ass'n") correlation module 246). The system memory 1010 may also include a synthesis project module 260 and an orchestrator interface module 262. One or more of the modules included in the memory 1010 may be configured to provide or consume output to implement one or more functions described herein.
[0151] In at least some examples, the structure and / or function of any of the above features can be implemented in software, hardware, firmware, circuits, or a combination thereof. Note that the above structures and constituent elements and their functions can be aggregated with one or more other structures or elements. Alternatively, if any, elements and their functions can be subdivided into constituent sub-elements. As software, various types of programming or formatting languages, frameworks, syntax, applications, protocols, objects, or technologies can be used to implement the above techniques. As hardware and / or firmware, various types of programming or integrated circuit design languages can be used to implement the above techniques, including hardware description languages, such as any register transfer language ("RTL") configured to design a field programmable gate array ("FPGA"), an application specific integrated circuit ("ASIC"), or any other type of integrated circuit. According to some embodiments, the term "module" may refer to, for example, an algorithm or a part of an algorithm, and / or a logic or a combination thereof implemented in hardware circuits or software. These can vary and are not limited to the examples or descriptions provided.
[0152] In some embodiments, the media content management system or one or more of its components or any process or device described herein may be in communication (e.g., wired or wireless) with, or may be located in, a mobile device such as a mobile phone or a computing device.
[0153] In some cases, a mobile device or any networked computing device (not shown) communicating with the action reminder controller or one or more components thereof (or any process or device described herein) can provide at least some of the structures and / or functions of any features described herein. As depicted in the above-mentioned figures, the structures and / or functions of any of the above-mentioned features can be implemented in software, hardware, firmware, circuits, or any combination thereof. Note that the above structures and constituent elements and their functions can be aggregated or combined with one or more other structures or elements. Alternatively, if any, elements and their functions can be subdivided into constituent sub-elements. As software, various types of programming or formatting languages, frameworks, grammars, applications, protocols, objects, or technologies can be used to implement at least some of the above-mentioned technologies. For example, at least one of the elements depicted in any of the figures can represent one or more algorithms. Alternatively, at least one of the elements can represent a part of logic, which includes a part of hardware configured to provide a constituent structure and / or function.
[0154] For example, the dynamic keyboard presentation module 212 or any one or more of its components or any process or device described herein may be implemented in one or more computing devices (i.e., any mobile computing device, such as a wearable device, an audio device (such as headphones or headsets), or a mobile phone, whether they are worn or carried) that include one or more processors configured to execute one or more algorithms in memory. Thus, at least some of the elements in the above figures may represent one or more algorithms. Alternatively, at least one of the elements may represent a portion of logic that includes a portion of hardware configured to provide a constituent structure and / or functionality. These may vary and are not limited to the examples or descriptions provided.
[0155] Additional disclosure
[0156] The techniques discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and partitioning of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.
[0157] The above-described structures and techniques may be implemented as hardware and / or firmware using various types of programming or integrated circuit design languages, including hardware description languages, such as any register transfer language ("RTL") configured to design a field programmable gate array ("FPGA"), an application specific integrated circuit ("ASIC"), a multi-chip module, or any other type of integrated circuit.
[0158] For example, a media content management system may be implemented in one or more computing devices including one or more circuits, including one or more components, or any process or device described herein. Thus, at least one of the elements in the above figures may represent one or more components of hardware. Alternatively, at least one of the elements may represent a portion of logic, including a portion of circuitry configured to provide structure and / or functionality.
[0159] According to some embodiments, the term "circuit" may refer to, for example, any system including multiple components through which current flows to perform one or more functions, including discrete components and complex components. Examples of discrete components include transistors, resistors, capacitors, inductors, diodes, etc., while examples of complex components include memory, processors, analog circuits, digital circuits, etc., including field programmable gate arrays ("FPGAs"), application specific integrated circuits ("ASICs"). Therefore, a circuit may include a system of electronic components and logic components (e.g., logic configured to execute instructions (such as, for example, a set of executable instructions of an algorithm), and therefore a component of a circuit). According to some embodiments, the term "module" may refer to, for example, an algorithm or a portion thereof, and / or logic implemented in hardware circuits or software or a combination thereof (i.e., a module may be implemented as a circuit). In some embodiments, an algorithm and / or a memory storing an algorithm is a "component" of a circuit. Therefore, the term "circuit" may also refer to, for example, a system of components including an algorithm. These may vary and are not limited to the examples or descriptions provided.
[0160] Although the foregoing examples have been described in some detail for the purpose of clear understanding, the above-mentioned inventive technology is not limited to the details provided. There are many alternative ways to implement the above-mentioned inventive technology. The disclosed examples are illustrative rather than restrictive.
[0161] The foregoing description of embodiments of the present invention has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the invention to the precise form disclosed. Those skilled in the relevant art will appreciate that many modifications and variations are possible in light of the above disclosure.
[0162] Certain parts of this description describe embodiments of the present invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are often used by those skilled in the art of data processing to effectively communicate the essence of their work to other persons skilled in the art. Although these operations are described functionally, computationally, or logically, they should be understood to be implemented by computer programs or equivalent circuits, microcodes, etc. In addition, without loss of generality, it has also been demonstrated that it is sometimes convenient to refer to these arrangements of operations as modules. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.
[0163] Any steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. In one embodiment, the software module is implemented using a computer program product that includes a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the steps, operations or processes described.
[0164] Embodiments of the present invention may also relate to an apparatus for performing the operations described herein. The apparatus may be specially constructed for the desired purpose, and / or it may include a general-purpose computing device selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium, or in any type of medium suitable for storing electronic instructions that can be connected to a computer system bus. In addition, any computing system mentioned in the specification may include a single processor, or may be an architecture that uses multiple processor designs to increase computing power.
[0165] Embodiments of the present invention may also relate to products produced by the computing process described herein. Such products may include information resulting from the computing process, wherein the information is stored on a non-transitory tangible computer-readable storage medium, and may include any embodiment of a computer program product or other data combination described herein.
[0166] Finally, the language used in this specification is selected primarily for readability and illustrative purposes, and may not be selected to define or limit the subject matter of the present invention. Therefore, it is intended that the scope of the present invention is not limited by this detailed description, but rather by any claims issued based on the application herein. Therefore, the disclosure of the embodiments of the present invention is intended to be illustrative, not to limit the scope of the present invention, which is set forth in the appended claims.
[0167] Although the subject matter has been described in detail for various specific example embodiments of the subject matter, each example is provided by way of explanation rather than limitation of the present disclosure. Those skilled in the art, after obtaining an understanding of the foregoing, can easily produce changes, modifications and equivalents of such embodiments. Therefore, the subject matter disclosure does not exclude the inclusion of such modifications, modifications and / or additions to the subject matter, which will be apparent to those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment can be used together with another embodiment to produce yet another embodiment. Therefore, it is intended that the present disclosure covers such changes, modifications and equivalents.
Claims
1. A computer-implemented method for extracting time information, include: Receiving data describing a media content item through a computing system including one or more computing devices, the media content item including a plurality of image frames for sequential display, wherein the plurality of image frames include: a first image frame and a second image frame, the first image frame including a first text string, the second image frame for sequential display after the first image frame, the second image frame including a second text string; inputting, by the computing system, data describing the media content item into a machine learning temporal analysis model, the machine learning temporal analysis model being configured to receive the data describing the media content item and, in response to receiving the data describing the media content item, output temporal analysis data describing temporal information associated with sequentially viewing a plurality of image frames of the media content item; and The time analysis data is received by the computing system as an output of the machine learning time analysis model, wherein the time information described by the time analysis data describes a semantic meaning described by sequentially reading the first text string before the second text string, and the semantic meaning is not described by reading the first text string or the second text string alone.
2. The computer-implemented method of claim 1, in, The time information is not described by individual image frames among the plurality of image frames.
3. The computer-implemented method of claim 1, in, One or more of the first text string and the second text string includes a single word without additional words or a single letter without additional letters.
4. The computer-implemented method of claim 1, in, The first text string of the first image frame has an appearance different from an appearance of the second text string of the second image frame in at least one of color, thickness, position, or font.
5. The computer-implemented method of claim 4, in, The temporal analysis data describes a semantic meaning associated with the first text string having an appearance different from an appearance of the second text string.
6. The computer-implemented method of claim 1, in: The plurality of image frames include a first image frame describing a first scene and a second image frame for sequential display after the first image frame, the second image frame describing a second scene; as well as The temporal information described by the temporal analysis data describes a semantic meaning described by viewing the first scene sequentially before the second scene, which semantic meaning is not described by viewing the first scene or the second scene alone.
7. The computer-implemented method of claim 1, in, The temporal analysis data describes the emotional content of the media content item.
8. The computer-implemented method of claim 1, further comprising: include: A content tag is assigned to the media content item based on temporal information described by the temporal analysis data.
9. The computer-implemented method of claim 1, in, The media content item includes an advertisement.
10. The computer-implemented method of claim 9, in, The temporal information described by the temporal analysis data describes the emotional content of the advertisement.
11. The computer-implemented method of claim 1 , further comprising: include: receiving, by a user computing device of the computing system, a search query for a media content item; selecting, by the computing system, the media content item based on the temporal analysis data describing temporal information; as well as The media content item is provided by the computing system in response to the search query.
12. The computer-implemented method of claim 11 , further comprising: include: A dynamic keyboard interface is provided for display by a user computing device of the computing system, and wherein the search query is received within the dynamic keyboard interface provided by the user computing device.
13. A computing system, include: one or more processors; a machine learning temporal analysis model configured to receive data describing a media content item and, in response to receiving the data describing the media content item, output temporal analysis data describing temporal information associated with sequentially viewing a plurality of image frames of the media content item, and wherein the temporal information is not described by individual image frames of the plurality of image frames; one or more non-transitory computer-readable media collectively storing instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: Receiving data describing a media content item including a plurality of image frames for sequential display, wherein the plurality of image frames include: a first image frame and a second image frame, the first image frame including a first text string, the second image frame for sequential display after the first image frame, the second image frame including a second text string; inputting data describing the media content item into the machine learning temporal analysis model; and The time analysis data is received as an output of the machine learning time analysis model, wherein the time information described by the time analysis data describes a semantic meaning described by sequentially reading the first text string before the second text string, and the semantic meaning is not described by reading the first text string or the second text string alone.
14. The computing system of claim 13, in, One or more of the first text string and the second text string includes a single word without additional words or a single letter without additional letters.
15. A computing system according to claim 13 or claim 14, in, The first text string of the first image frame has an appearance different from an appearance of the second text string of the second image frame in at least one of color, thickness, position, or font.
16. The computing system of claim 15, in, The temporal analysis data describes a semantic meaning associated with the first text string having an appearance different from an appearance of the second text string.
17. The computing system of claim 13, in, The media content item includes an advertisement, and wherein the temporal information described by the temporal analysis data describes an emotional content of the advertisement.
18. A computer-implemented method for training a machine learning temporal analysis model, the method include: Receiving data describing a media content item through a computing system including one or more computing devices, the media content item including a plurality of image frames for sequential display, wherein the plurality of image frames include: a first image frame and a second image frame, the first image frame including a first text string, the second image frame for sequential display after the first image frame, the second image frame including a second text string; inputting, by the computing system, data describing the media content item into the machine-learned temporal analysis model, the machine-learned temporal analysis model being configured to receive the data describing the media content item and, in response to receiving the data describing the media content item, output temporal analysis data describing temporal information associated with sequentially viewing a plurality of image frames of the media content item, and wherein the temporal information is not described by individual image frames of the plurality of image frames; receiving, by the computing system, the temporal analysis data as an output of the machine learning temporal analysis model, wherein the temporal information described by the temporal analysis data describes a semantic meaning described by sequentially reading the first text string before the second text string, the semantic meaning not described by reading the first text string or the second text string alone; and One or more parameters of the machine learning temporal analysis model are adjusted by the computing system based on a comparison of the temporal analysis data with benchmark truth temporal analysis data.
19. A computing system, include: at least one processor; At least one tangible, non-transitory computer-readable medium storing instructions which, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 12.
20. A computer readable medium storing instructions which, when executed by at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 12.
