Systems and methods of determining semantic meaning of content based on image analysis

Abstract

A system may include a processor programmed to access content comprising visual data, determine, based on a language model trained using autoregression to predict words or phrases in text, a topic based on words or phrase in the content, identify a plurality of scenes from the visual data, each scene from among the plurality of scenes occupying a respective length of time in the visual data, select, for each scene from among the plurality of scenes, at least one frame in the scene, resulting in a plurality of selected frames from the content, wherein the plurality of selected frames comprises at least one frame from each scene, generate an image wall based on the plurality of selected frames, activate a computer vision model based on the topic and the image wall; and determine a semantic meaning of the content based on the activation of the computer vision model.

Description

Systems and Methods of Determining Semantic Meaning of Content Based on Image AnalysisBACKGROUND

[0001] Systems for identifying a semantic meaning of natural language may involve understanding the relationships between words in text. For example, language models may be trained on large training datasets of text to understand the structure and relationships of words. This learned understanding allows these systems to understand the semantic meaning of words in text. However, these systems are unable to obtain a semantic understanding of visual data such as video or images.BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Features of the present disclosure may be illustrated by way of example and not limited in the following figure(s), in which like numerals indicate like elements, in which:

[0003] FIG. 1 shows an illustrative system environment for determining semantic meaning of content based on image analysis, according to an implementation.

[0004] FIG. 2 illustrates a schematic dataflow for determining semantic meaning of content based on image analysis, according to an implementation.

[0005] FIG. 3 illustrates a schematic dataflow for determining semantic meaning of content based on image analysis using an image wall with tiling by a computer vision model, according to an implementation.

[0006] FIG. 4 illustrates an example method of determining semantic meaning from visual data, according to an implementation.DETAILED DESCRIPTION

[0007] Various systems and methods may determine semantic meaning of content based on image analysis of visual data in content. The content may include any visual data such as video or images and be associated with words or phrases in the content. For purposes of illustration, the content may be described herein throughout as a video file or stream having a video component (also referred to as simply “video,’’ which is an example of visual data) and an audio component (audio data) in which a transcript of the audio data is available. The system may access the transcript (such as closed captioning data or other transcription) or may generate the transcript using automated speech recognition from the audio data.

[0008] The system may determine a semantic meaning of the content based on image analysis of the video using a deep learning computer vision model trained to analyze images. For example, the sy stem may determine the semantic meaning of a video file or stream based on image analysis of the video component. In particular, the system may identify a plurality of scenes from the video based on scene transitions. The sy stem may select a frame from each scene resulting in a plurality of selected frames that includes at least one frame from each scene. The system may pre-process each frame such as by cropping, scaling, and enhancing each frame. The system may stitch the pre-processed frames into an image wall, which is a composite image of the pre-processed frames. Each frame in the image wall will occupy its own discrete position in the image wall.

[0009] To provide context for image analysis, the system may access a transcript of the content and determine a topic of the content based on the transcript. The transcript may be a transcription of one or more words or phrases from the content. The words or phrases may include dialogue, narration, and / or other words or phrases associated with or otherwise appearing in the content. The transcript may have been previously generated or may be generated by the system for topic identification. For example, the content may be video file or stream that includes narration or dialog relating to various aspects of video technology. The system may use a deep learning language model trained to understand relationships between words to identify the topic from the transcript. For example, the language model may determine that the content, based on the words or phrases used in the content, relates to “video technology.”

[0010] The system may then activate a computer vision model using the image wall and the topic. The computer vision model may return determine a semantic meaning of the content based on image analysis of the image wall and the contextual information provided by the topic.

[0011] FIG. 1 shows an illustrative system environment 100 for determining semantic meaning of content based on image analysis, according to an implementation. The system environment 100 may include one or more client devices 104, a computer system 110, and a model Application Programming Interface (“API”) endpoint 111. The content 102 is a stream, a file, and / or other type of electronic data. The content 102 may include visual data, audio data, and / or other type of data (such as haptic data). Examples of content 102 include a video, images, slide presentation, and / or other types of content. The content 102 may include spoken words, written words, and / or other sounds. Each client device 104 is a device that may be used by an end user to interact with the computer sy stem 1 10. For example, each client device 104 may include a desktop computer, laptop computer, tablet computer, smartphone, and / or other types of devices that may communicate with the computer sy stem 110.

[0012] The model API endpoint 111 is an API that provides an interface to one or more of the models. Although only one endpoint is shown, there may be multiple endpoints that each interface with a respective model. The system may activate a model via the model API endpoint 111. For example, to activate a model, the computer system 110 may generate or select a prompt via the prompt generation system 140 and transmit the prompt as input (along with any other inputs described herein) via the model API endpoint 111. For models that do not expose an API, the various systems of the computer system 110 may call the model directly through interfaces (such as a command line interface) exposed by the model.

[0013] The computer system 110 is a computational platform having one or more computer devices that determines a semantic meaning of the content 102 based on image analysis. The computer system 110 may include a processor 112, a scene detection system 120, a topic identification system 130, a prompt generation system 140, an image wall generation system 150, an image analysis system 160, and / or other features. The computer system 110 may access (such as read, write, delete, and / or update) variousdatabases, such as the content repository 101, the prompt repository 103, and training repository 105. The content repository 101 may store the content 102. The prompt repository 103 may store previously generated prompts for re-use. The training repository' 105 may store training data for training one or more of the models.

[0014] The computer system 110 may train, retrain, fine-tune, execute, or otherwise activate various computer models. The computer models may include a language model 121, a vision model 125, and / or other models. At least some of these models are generative Al models. A generative Al model is a computer model that is trained to generate new content based on training data.

[0015] Each of the systems 120, 130, 140, 150, and 160 may call or otherwise use one or more of the other systems. For example, the image analysis system 160 may call the prompt generation system 140 to generate a prompt for a computer vision model 125. Similarly, each of the systems 120, 130, 140, 150, and 160 may train, retrain, fine-tune, execute, and / or otherwise activate various computer models such as models 121 and 125.

[0016] The processor 112 may include one or more of a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and / or other mechanisms for electronically processing information. Although processor 112 is shown in FIG. 1 as a single entity, this is for illustrative purposes only. In some implementations, processor 112 may comprise a plurality of processing units. These processing units may be physically located within the same device, or processor 112 may represent processing functionality of a plurality of devices operating in coordination. Some or all processing units may be on-site within a computational facility and / or be located remotely such as at a cloud-based computing facility

[0017] Computer Models, including Generative Al Models

[0018] The language model 121 is a generative Al model for language. In particular, the language model 121 may be a pretrained deep-learning Large Language Model (LLM) trained on large language datasets. The language model 121 may be trained to semantically understand natural language and automatically generate new text based on this understanding. Examples of the language model 121 may include, withoutlimitation, one or more variants of: OpenAI GPT, LLaMA from META, Google LaMBDA, BERT from GOOGLE, BigScience BLOOM, Multitask Unified Model (MUM), or other language models.

[0019] The computer system 110 may activate the language model 121 with one or more input prompts and one or more model parameter values. The prompt may be generated by the prompt generation system 140. A model parameter value is an input that specifies behavior - and therefore output - of the language model 121. For example, a model parameter value may include a temperature parameter that adjusts the level of randomness for automatically generated text. Different temperature parameter values will result in different levels of randomness in the generated text. Thus, the temperature parameter value may be used to control the output of the language model 121. The language model 121 may return the automatically generated text based on the one or more input prompts and any model parameter values.

[0020] The computer vision model 125 is a generative Al model that is trained to perform image analysis on visual data. For example, the computer vision model 125 is trained to process, understand, and identify objects in electronic visual data. As used herein, the term “visual data” will generally refer to images and video, whether two dimensional (2D) or three dimensional (3D). Computer vision models may use Convolutional Neural Networks (CNNs). CNNs may break down an image into a smaller grids of pixels. These models may apply filters to these grids, identifying edges, shapes, and patterns. Through multiple layers, the model combines and analyzes these features, gradually building a higher-level understanding of the image. The model may then classify the image or detect one or more objects within the image. Thes models may be trained based on large training sets comprising labeled with the objects in the images. By analyzing millions of labeled images, the model learns to recognize patterns and connect them to specific objects or concepts. The computer vision model 125 may be trained to perform various computer vision tasks, such as classification, object detection, segmentation, and / or other tasks.

[0021] In some implementations, a computer vision model 125 may implement transformers. Transformers analyze the entire image at once, focusing on relationships between different parts. This allows them to capture long-range dependencies in complex imagery’ for image classification, object detection, image generation, and video understanding. Vision transformers may partition the image intopatches, then use the transformer architecture to analyze the relationships between these patches, learning the global context of the image.

[0022] Examples of the computer vision model 125 includes the GPT-4 vision model, LaVA (Large Language and Vision Assistant), and BakLLaVA. These or other computer vision models 125 may integrate image identification and language understanding that provides an ability to analyze visuals and ask questions of the visual data.

[0023] Scene Detection

[0024] The scene detection system 120 may identify a plurality of scenes in the content 102. For example, the scene detection system 120 may identify the plurality of scenes by identifying scene transitions in the content 102. A scene transition is a separator between scenes of the content 102. Thus, scene transitions define individual scenes in the content 102. A scene is a discretely identifiable portion of the content 102. For example, if the content 102 is a video, a scene transition may separate one scene (part of the video) from the next scene in the video. If the content 102 is a slideshow presentation, a scene transition may separate one slide from the next slide. In some of these examples, a scene transition may occur within a slide, such as when animations are present within a single slide. In this instance, a scene transition may occur between one part of the slide to a next part of the slide.

[0025] Scene transitions and scenes may occur along discrete sections of the content 102, such as along a timeline in a video, slide number in a slide presentation, or a timeline in spoken presentation associated with the slide presentation. In some instances, the audio data in the content 102 may include dialog, narration, and / or other spoken words that are typically, though not necessarily, natural language in nature. For instance, if the content 102 is a video, then the audio data may include dialog or narration within the video. If the content 102 is a slide presentation, the audio data may include a person’s spoken presentation that accompanies the slide presentation and / or words within the slide presentation itself. In some instances, a transcript of spoken words in the audio data may be obtained and / or generated by the scene detection system 120. If generated, the scene detection system 120 may use an audio model such as an automatedspeech recognition (“ASR”) system that converts audio into text. The scene detection system 120 may break down video into individual frames. Each frame is an image taken from the sequence of video. In some implementations, the scene detection system 120 may convert the frames from color to grayscale to focus on intensity variations rather than color information for scene changes. The scene detection system 120 may extract one or more characteristics from each frame that can be used to compare frames to identify differences between the frames. The comparison may include, without limitation, histogram analysis, edge detection, block-based methods, and optical flow. In histogram analysis, histograms capture the distribution of pixel intensities in an image. Significant changes in the histogram between frames can indicate a scene transition. In edge detection, edges may correspond to boundaries between objects in a scene. By comparing edge maps of consecutive frames, significant changes that may indicate a scene transition may be detected. In block -based methods, the image is divided into smaller blocks, and features such as average intensity or standard deviation arc calculated for each block. Comparing these features between frames helps identify areas of significant change, and therefore a potential scene change. In optical flow, motion patterns between frames are analyzed. Sudden shifts in motion patterns may be indicative of scene transitions.

[0026] In some implementations, once these features are extracted, various algorithms such as thresholding or statistical methods may compare the features and determine whether a scene transition has occurred. In thresholding, a threshold is set for the difference between features of consecutive frames. If the difference exceeds the threshold, a potential scene transition is detected. In statistical methods, statistical models may be used to analyze feature variations and identify deviations that are more likely to indicate scene transitions.

[0027] Topic Identification

[0028] The topic identification system 130 may identify words or phrases for the content 102 and determine a topic 131 based on the words or phrases. For example, if the content 102 is video, the topic identification system 130 may obtain a previously generated transcript of the video. Alternatively, or additionally, the topic identification system 130 may obtain a transcript of the spoken words in the video based on Automated Speech Recognition (ASR). If the content 102 is a slide presentation, the topicidentification system 130 may obtain a transcript of the slide presentation, such as a transcript of a spoken presentation. Alternatively, or additionally, the topic identification system 130 may obtain other words or phrases associated with the slide presentation, such as embedded audio or words written in the notes section or other metadata. In either instance, the topic identification system 130 may obtain words or phrases that are typically, though not necessarily, natural language associated with the content 102, whether dialog, narration, and / or other words of phrases associated with the content. It should be noted that the content 102 may include a video recording of a slide presentation in which slides are presented for visual data and the video recording includes spoken words by a presenter during slide presentation.

[0029] The topic identification system 130 may leverage autoregression to understand natural language and identify topics conveyed in words or phrases, such as natural language. Autoregression is a statistical approach for modeling sequences of data such as text. Autoregression predicts the next item in a sequence based on the previous items. Autoregression assumes that the current value in a sequence is related to its preceding values. To capture these relationships, autoregression uses lagged variables. These are the previous values in a sequence. Autoregression may use a variant of linear regression. In autoregression, the current value is the dependent variable being predicted, and the lagged variables are the independent variables. The model learns the weights assigned to each lagged variable to best predict the current value. Once the model is trained on historical data, the model may predict future values given an input. For example, a language model trained using autoregression on a large training data of text content may learn to predict the next words given a set of input words. In particular, in the context of language, for example, autoregression predicts the next word in a sentence based on previous words in the sentence (lagged variables). Autoregression may be used to train the language model 121 using large language training data. From this large language training data, the language model 121 may be trained to predict next words in a sequence and to combine this knowledge to generate text. In the context of topic identification, the language model 121 may use this trained understanding of text to generate words or phrases that are likely to reflect the topic of an input sequence of words or phrases, such as text from a scene.

[0030] This understanding of language may be used to identify topics in words or phrases. For topic identification, the model may identify the subject of text such as a transcript of the content 102 based on the sequence of words in the transcript. The language model 121 may capture the statistical relationships between words in a transcript. This information can then be fed into other algorithms, like hidden Markov models (HMMs) or recurrent neural networks (RNNs), which are specifically designed for topic identification. These algorithms can use the word dependencies captured by AR to learn the underlying themes in the transcript.

[0031] Dynamic Prompt Generation Based on Content Topics

[0032] The prompt generation system 140 is a system component that receives an input and generates a prompt for execution by one or more of the models, such as the language model 121 or the computer vision model 125. A prompt is an instruction to a generative Al model to generate an output. The prompt may include a query to be answered and / or a description of the output to be generated. In some instances, the prompt may also include additional information to be used by the model to generate a response. The additional information may include contextual data, desired output formats, constraints, domain-specific knowledge, examples, templates, tone, style, localization information (such as output language, consideration of cultural information, and so forth), and / or other information that may be provided to the model to help shape its response. Thus, generation of the prompt itself can be an important factor in obtaining an appropriate response from one or more of the generative Al models.

[0033] The prompt generation system 140 may generate a prompt for a language model 121 to identify topics of words or phrases. For example, the topic identification system 130 may provide the prompt generation system 140 with words or phrases to generate a prompt that requests the language model 121 to identify the topic based on the transcript of and / or other words or phrases from the content 102.

[0034] The image analysis system 160 may provide the prompt generation system 140 with the image wall from the scene to generate a prompt that requests the computer vision model 125 to identify a semantic meaning of the scene based on the image and the topic of the scene.

[0035] Image Stitching and Identification for Image Analysis

[0036] The image wall generation system 150 may select one or more frames (images) from each scene. The image wall generation system 150 may select a frame based on its position in the scene and / or other selection criteria. For example, the image wall generation system 150 may select a frame in the middle of the scene (if a scene has 10 frames, frame five or six may be selected or if the scene is 10 seconds long, then a frame at the 5 or 6 second mark may be selected). Other frames such as the first or last frame may be selected. Other selection criteria may include selecting the frame based on motion, color, brightness, and / or other characteristics. Each selected frame may be pre-processed by such as by resizing, cropping, enhancing, or otherwise preparing the frame for image analysis.

[0037] The image wall generation system 150 may generate an image wall 151 based on the selected frames. An image wall 151 is a composite image that is composed of two or more individual images combined in a way that each individual image occupies a discrete portion of the composite image. For example, an image wall 151 is a composite of the selected frames. The image wall 151 may include the frames tiled in a grid configuration. The grid configuration is an arrangement having one or more columns and one or more rows in which an individual image occupies a respective row and column. For example, in an image wall 151 having a 1x4 grid configuration (one column by four rows), four individual images are tiled together such that an individual image occupies a respective row in a single column. In a 2x4 grid configuration, eight individual images arc tiled together across two columns (with four images in each column, one image per each row). Other grid configurations and other arrangements of individual images may be used to form the composite image. Based on the grid configuration or other configuration, image processing analysis may be performed on each individual image based on their location in an image wall 151 . For example, depending on the resolution of the individual images and therefore the composite image, each individual image may be located according to its pixels in the grid or other configuration.

[0038] The image wall generation system 150 may output the image wall 151 in various formats suitable for input to the computer vision model 125. For example, the image wall generation system 150 may output the image wall 151 in a Base64 encoding or generate an image file in one of various image file formats (such as JPEG, PNG, etc.). The Base64 encoding (or other encoding) may be transmitted directly to thecomputer vision model 125 for image processing. The image file may be stored at a Uniform Resource Location (URL), which is transmitted to the computer vision model 125, which accesses the image file at the URL for image processing.

[0039] Semantic Understanding of Content Based on Image Analysis

[0040] An image analysis system 160 may determine a semantic meaning of the content 102 based on image analysis of visual data from the content. For example, the image analysis system 160 may use the prompt generation system 140 to generate a prompt that includes one or more topics from each scene determined by the topic identification system 130. The image analysis system 160 may activate the computer vision model 125 with the prompt and the image wall 151. The computer vision model 125 may be trained to analyze the visual data in the image wall 151, which may be tiled, and generate a requested output based on the visual data and any prompts used to activate the computer vision model 125. Tn this case, the prompt may request a semantic summary of the visual data along with one or more topics to provide the computer vision model 125 with contextual information to determine the semantic summary of the visual data.

[0041] Example operational flows

[0042] Having described an overview of system functions, an example operation of the computer system 110 will be described with reference to FIG. 2. FIG. 2 illustrates a schematic dataflow' 200 for determining semantic meaning of content based on image analysis, according to an implementation. The dataflow 200 shows an example operational implementation of the system environment 100 illustrated in FIG. 1. As such, reference numbers in FIGS. 1 and 2 will refer to the same system features.

[0043] In the illustrated example, content 102 is illustrated as a video that plays over a timeline T (starting at To and ending at T ' N). The scene detection system 120 may identify a plurality of scenes 202A-N from the video. Each scene 202 may include a plurality of frames, the number of which w ill vary depending on the bitrate, particular content in the video, and / or other factors. In some implementations, the scene detection system 120 may identify the scenes 202 A-F based on transitions 204A-E, each occurring at timeTA-TG along the timeline T. Although shown as equal points in time, each scene transition 204 may span a length of time that is the same or different than other scene transitions 204.

[0044] When a scene transition 204 is detected, the scene detection system 120 may determine that a prior scene has ended and a next scene will begin. For example, the scene detection system 120 may determine that scene 202A has ended or is ending when scene transaction 204A has been detected and that scene 202B will start or has started. Some or all of a given scene transition 204 may be included with the prior scene. Alternatively or additionally, some or all of a given scene transition 204 may be included with the next scene. For example, some or all of the scene transition 204A may be included with scene 204A and / or some or all of the scene transition 204A may be included with scene 204B.

[0045] The scene detection system 120 may keep track of when the scenes 202 occur along the timeline T so that content in the scene can be analyzed. For instance, the scene detection system 120 may determine that scene 202A begins at To and ends at TA. In this manner, frames may be selected from each scene 202 based on when they appear in the scene Likewise, words or phrases, such as spoken words or phrases in dialogue or narration during the video, may be obtained for each scene. As previously noted, these words or phrases may be part of a previously generated transcript (such as from a video producer), ASR recognition, and / or other source for a transcription of the audio.

[0046] The topic identification system 130 may determine a topic of the content 102 based on words or phrases associated with the content. For example, the topic identification sy stem 130 may use the prompt generation system 140 to generate a prompt for a language model 121 to determine a topic from the words or phrases. The words or phrases may be a transcript of the content 102 and / or other words or phrases associated with the content.

[0047] The image wall generation system 140 may generate an image wall 105 from each frame selected from each scene. The image analysis system 150 may generate a semantic summary of the content 102 based on image analysis of the image wall 105 and the topic identified from the content.

[0048] FIG. 3 illustrates a schematic dataflow for determining semantic meaning of content 102 based on image analysis using an image wall 105 with tiling by a computer vision model 125, according to animplementation. The dataflow 300 shows an example operational implementation of the system environment 100 illustrated in FIG. 1. As such, reference numbers in FIGS. 1 and 3 will refer to the same system features.

[0049] The scene detection system 120 may detect a plurality of scenes 302A-F, which may be identified based on respective scene transactions 304A-E, such as described in FIG. 2. The scene detection system 120 may select one or more frames 303A-F from each corresponding scene 302A-F.

[0050] Some computer vision models 125 have constraints on the input image. For example, a computer vision model 125 may have a maximum input resolution per individual input image. If an input image exceeds the maximum input resolution, the computer vision model 125 may break apart the input image into tiles that are each within the maximum input resolution and then perform image processing on each tile.

[0051] To address this issue, the image wall generation system 150 may generate the image wall 151 to conform to these constraints. For example, the image wall generation system 150 may pre-process individual input images to minimize tile generation by the computer vision model 125. This will reduce processing complexity and compute time. To do so, the image wall generation system 150 may crop, scale and sharpen each of the individual images to fit within at least one dimension of the maximum input resolution.

[0052] To illustrate, the maximum input resolution of a computer vision model 125 may be, without limitation, 512x512 (pixels). A frame 303A selected from scene 302A may have a resolution of 720P (720x1280). Other selected frames 303B-Gmay have similar resolutions. The image wall generation system 150 may pre-process frame 303A to reduce the size of the frame 303A. Such reduction may be along the height and / or vertical axis. For instance, the image wall generation system 150 may crop and / or scale the frame 303A so that at least one of the dimensions of the frame 303A is within the maximum input resolution.

[0053] As illustrated, the image wall generation system 150 may crop and / or scale the frame 303A along the horizontal dimension to fit or be below the maximum horizontal dimension of 512 pixels (reduction from 1280 to 512 in this case). Alternatively, or additionally, the image wall generation system 150 maycrop and / or scale the frame 303A along the vertical dimension to fit the maximum vertical dimension of 512 pixels (reduction from 720 to 300 in this case). It should be noted that when both dimensions are reduced, they may maintain the original ratio of the original frame or not maintain the original ratio. As illustrated, the frame 303A is cropped and scaled from an original 720x1280 resolution to 512x300 resolution. The image wall generation system 150 may sharpen the frame 303 A, such as by enhancing the edges or details of the re-sized frame. Various techniques for sharpening may be used, such as unsharp masking, high-pass filtering, Laplacian sharpening, and / or other techniques to sharpen an image. The image wall generation system 150 may crop, scale, and / or sharpen the other selected frames 303B-G as well.

[0054] The image wall generation system 150 may stitch the frames 303A-G (which may each have been pre-processed by resizing and sharpening) into an image wall 151. Stitching may be sequentially performed in a grid-like pattern using a single column as illustrated, in which the pre-processed frame 303 A is placed on top, followed by frames 303B-G. Two or more columns may be used instead, in which case the frames may be placed in the left-most column until a predetermined number of rows are filled, followed by the next column, and so forth until the frames 303 have been stitched together. Other patterns of filling the grid-like pattern may be used as well.

[0055] The image analysis system 160 may access the topic of tire content 102 identified by the topic identification system 130 and the image wall 151 generated by the image wall generation system 150 and determine a semantic meaning of the content 102 based on the topic and the image wall 151. In some instances, the image analysis system 160 may determine a semantic meaning of each scene of the content 102 based on the individual frames tiled in the image wall 151. In particular, the image analysis system 160 may activate the computer vision model 125 to generate a semantic meaning of the content 102 based on the topic and the image wall 151. To illustrate, in one particular example, the identified topic of the content 102 is “video technology.” The image analysis system 160 may generate a prompt (via the prompt generation system 140): “The image shows video frames in sequence. There are 6 frames. Please show the important tags specific for video technology." The image analysis system 160 may provide the computer vision model 125 with the prompt and the image wall 151 (such as via Base64 encoding, URL, or otherimage input used by the computer vision model 125). The computer vision model 125 will tile the image wall 151 based on the size of the image wall 151 and the maximum input resolution of the computer vision model 125. For instance, the image wall 151 is 512 x 1800 because six frames 512x300 each are tiled in a single row in the image wall 151. The computer vision model 125 may therefore generate four tiles 511A- D across the 6 frames. The computer vision model 125 may then output tags per frame as illustrated in Table 1 below.

[0056] Table 1. Example of tags per frame describing the frames tiled in the image wall 151.

[0057] Referring to Table 1, tags 1A-N represent tags, or keywords, that the computer vision model 125 has determined describe the image in frame 303 A. Likewise, tags 2A-N, tags 3A-N, tags 4A-N, tags 5A- N, and tags 6A-N are tags respectively determined by the computer vision model 125 from frames 303B- F. The number of tags for each frame may vary depending on processing by the computer vision model 125.

[0058] The image analysis system 160 may generate a semantic summary of the tags 1A-N through 6A- N. For example, the semantic summary' may include a listing of the tags. Alternatively or additionally, the image analysis system 160 may generate another prompt for the language model 121 to generate a semantic summary7based on the tags. The prompt in this example will include the tags and an instruction to generate a semantic summary' based on the tags, in which case the language model 121 may return the semantic summary'. In some of these examples, the prompt may include the topic for additional context. In some ofthese examples, the prompt may include some or all of the words or phrases from the content 102 (such as some or all of the transcript of the content 102).

[0059] FIG. 4 illustrates an example method 400 of determining semantic meaning from visual data, according to an implementation. At 402, the method 400 may include accessing content 102 comprising visual data.

[0060] At 404, the method 400 may include determining, based on a language model 121 trained using autoregression to predict words or phrases in text, a topic 131 of the content 102 based on words or phrase in the content.

[0061] At 406, the method 400 may include identifying a plurality' of scenes (such as scenes 202 or scenes 302) from the visual data, each scene from among the plurality' of scenes occupying a respective length of time in the visual data.

[0062] At 408, the method 400 may include selecting, for each scene from among the plurality of scenes, at least one frame (such as frame 303) in the scene, resulting in a plurality of selected frames from the content, wherein the plurality of selected frames comprises at least one frame from each scene.

[0063] At 410, the method 400 may include generating an image wall (such as image wall 151) based on the plurality' of selected frames.

[0064] At 412, the method 400 may include activating a computer vision model (such as the computer vision model 125) based on the topic and the image wall. For example, the computer vision model may output tags for each frame that is tiled in the image wall. An example of these tags is illustrated in Table 1.

[0065] At 414, the method 400 may include determining a semantic meaning of the content based on the activation of the computer vision model. For example, the semantic meaning may be based on the tags associated with each scene in the image wall.

[0066] Training the various models as disclosed herein may include supervised, semi-supervised, and unsupervised techniques. For example, when training models with labeled data, supervised machine learning techniques may be used. Suitable models may be used for this task such as supervised learning models, rule-based systems. Conditional Random Fields (CRFs), Recurrent Neural Networks (RNNs) andLong Short-Term Memory' (LSTMs). Deep Learning Models such as transformers including models such as BERT and RoBERTa, has shown excellent performance in various NLP tasks, including text parsing. Model Training: Splitting Data: Divide your labeled data into training, validation, and testing sets. The training set is used to train the model, the validation set helps fine-tune hyperparameters, and the testing set evaluates the model’s final performance on unseen data. Hyperparameter Tuning may involve adjusting hyperparameters (learning rate, batch size, etc.) of the chosen model to optimize its performance on the validation set.

[0067] The model leams to identify patterns and relationships within the labeled data to predict the corresponding JSON structure for new, unseen text. Evaluating the model’s performance on the testing set may be performed based on metrics such as accuracy (percentage of correctly parsed documents) and Flscore (harmonic mean of precision and recall). Analyzing errors made by the model and identifying areas for improvement may involve collecting more labeled data or refining the model architecture / hyperparameters.

[0068] To ingest the content 102, the computer system 110 may use a system API (not illustrated) to provide upload capabilities for client devices 104. This data upload or access may be made via Java Database Connectivity (JDBC), Representational state transfer (RESTful) services. Simple Mail Transfer Protocol (SMTP) protocols, direct file upload, and / or other file transfer sendees or techniques. In particular, the system API 113 may include a MICROSOFT SHAREPOINT API Connector, an Hyper Text Transfer Protocol (HTTP)ZHTTP-secure (HTTPS), a Network Drive Connector, a File Transfer Protocol (FTP) Connector, SMTP Artifact Collector, Object Store Connector, MICROSOFT ONEDRIVE Connector, GOOGLE DRIVE Connector, DROPBOX Connector, and / or other types of connector interfaces.

[0069] The computer system 110 and the one or more client devices 104 may be connected to one another via a communication network (not illustrated), such as the Internet or the Internet in combination with various other networks, like local area networks, cellular networks, or personal area networks, internal organizational networks, and / or other networks. It should be noted that the computer system 110 may transmit data, via the communication network, conveying the data to one or more of the client devices 104.The data may be conveyed via a user interface generated for display al the one or more client devices 104, one or more messages transmitted to the one or more client devices 104, and / or other types of data for transmission. Although not shown, the one or more client devices 104 may each include one or more processors.

[0070] Processor 112 may be programmed to execute one or more computer program components. The computer program components may include software programs and / or algorithms coded and / or otherwise embedded in the processor 112. The one or more computer program components or features may include various subsystems such as the , and / or other components.

[0071] Processor 112 may' be configured to execute or implement 120, 130, 140, 150, and 160 by' software; hardware; firmware; some combination of software, hardware, and / or firmware; and / or other mechanisms for configuring processing capabilities on processor 1 12. It should be appreciated that although 120, 130, 140, 150, and 160 arc illustrated in FIG. 1 as being co-locatcd in the computer system 110, one or more of the components orfeatures 120, 130, 140, 150, and 160 may be located remotely from the other components or features. The description of the functionality provided by the different components or features 120, 130, 140, and 150 described below is for illustrative purposes, and is not intended to be limiting, as any of the components or features 120, 130, 140, 150, and 160 may provide more or less functionality than is described, which is not to imply that other descriptions arc limiting. For example, one or more of the components or features 120, 130, 140, 150, and 160 may' be eliminated, and some or all of its functionality may be provided by others of the components or features 120, 130, 140, 150, and 160 again which is not to imply that other descriptions are limiting. As another example, processor 112 may include one or more additional components that may perform some or all of the functionality' attributed below to one of the components or features 120, 130, 140, 150, and 160.

[0072] Each of the computer system 110 and client devices 104 may also include memory in the form of electronic storage. The electronic storage may include non-transitory storage media that electronically stores information. The electronic storage media of the electronic storages may include one or both of (i) system storage that is provided integrally (e.g., substantially non-removable) with serv ers or client devicesor (ii) removable storage that is removably connectable to the servers or client devices via, for example, a port (e.g., a USB port, a firewire port, etc.) or a drive (e.g., a disk drive, etc.). The electronic storages may include one or more of optically readable storage media (e g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.), electrical charge-based storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.), and / or other electronically readable storage media. The electronic storages may include one or more virtual storage resources (e.g., cloud storage, a virtual private network, and / or other virtual storage resources). The electronic storage may store software algorithms, information determined by the processors, information obtained from servers, information obtained from client devices, or other information that enables the functionalities described herein.

[0073] The databases and data stores (such as 101 , 103, 105) may be, include, or interface to, for example, an Oracle™ relational database sold commercially by Oracle Corporation. Other databases, such as Informix™, DB2 or other data storage, including file-based, or query formats, platforms, or resources such as OLAP (On Line Analytical Processing), SQL (Structured Query' Language), a SAN (storage area network), Microsoft Access™ or others may also be used, incorporated, or accessed. The database may' comprise one or more such databases that reside in one or more physical devices and in one or more physical locations. The database may include eloud-based storage solutions. The database may' store a plurality' of types of data and / or files and associated data or file descriptions, administrative information, or any other data. The various databases may store predefined and / or customized data described herein.

[0074] The systems and processes are not limited to the specific implementations described herein. In addition, components of each system and each process can be practiced independently and separate from other components and processes described herein. Each component and process also can be used in combination with other assembly packages and processes. The flow charts and descriptions thereof herein should not be understood to prescribe a fixed order of performing the method blocks described therein. Rather the method blocks may be performed in any' order that is practicable including simultaneousperformance of at least some method blocks. Furthermore, each of the methods may be performed by one or more of the system features illustrated in FIG. 1.

[0075] This written description uses examples to disclose the implementations, including the best mode, and to enable any person skilled in the art to practice the implementations, including making and using any devices or systems and performing any incorporated methods. The patentable scope of the disclosure is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal language of the claims.

Claims

What is claimed is:

1. A computer system, comprising: a processor programed to. access content comprising visual data; determine, based on a language model trained using autoregression to predict words or phrases in text, a topic of the content based on words or phrase in the content; identify a plurality of scenes from the visual data, each scene from among the plurality of scenes occupying a respective length of time in the visual data; select, for each scene from among the plurality of scenes, at least one frame in the scene, resulting in a plurality of selected frames from the content, wherein the plurality of selected frames comprises at least one frame from each scene; generate an image wall based on the plurality of selected frames; activate a computer vision model based on the topic and the image wall; and determine a semantic meaning of the content based on the activation of the computer vision model.

2. The system of claim 1, wherein to identify the plurality of scenes, the processor is further programmed to: identify a plurality of scene changes, each scene change indicating a change from one scene to another scene along a timeline of the content.

3. The system of claim 1, wherein the visual data comprises video, each scene representing a portion of the video, and wherein to, for each scene, identify at least one image in the scene, the processor is programmed to select at least one frame from the scene, the at least one frame corresponding to the at least one image.

4. The system of claim 3, wherein the processor is further programmed to: crop and / or scale the at least one image to reduce the size of the at least one image based on an image size limit of a computer vision model.

5. The system of claim 4, wherein the computer vision model has a maximum horizontal pixel dimension and a maximum vertical pixel dimension, and wherein to crop or scale the at least one image, the processor is further programmed to:(1) crop and / or scale the at least one image to reduce at least the horizontal dimension of the at least one image based on the maximum horizontal dimension, and / or (2) crop and / or scale the at least one image to reduce at least the vertical dimension of the at least one image based on the maximum vertical dimension.

6. The system of claim 3, wherein the processor is further programmed to: stitch each image in the image wall to conform to a resolution requirement of the image model.

7. The system of claim 1, wherein the processor is further programmed to: access a transcript of the content, the transcript being a transcription of spoken or written words of tlie content, wherein the one or more words or phrases are accessed from the transcript.

8. The system of claim 7, wherein to identify the topic, the processor is further programmed to: perform autoregression analysis on the transcript.

9. The system of claim 1, wherein the computer vision model generates a listing of tags per image in the image wall and wherein the processor is further programmed to: determine semantic meaning of the visual data based on the listing of tags per image.

10. A method, accessing, by a processor, content comprising visual data; determining, by the processor, based on a language model trained using autoregression to predict words or phrases in text, a topic of the content based on words or phrase in the content; identifying, by the processor, a plurality of scenes from the visual data, each scene from among the plurality of scenes occupying a respective length of time in the visual data; selecting, by the processor, for each scene from among the plurality of scenes, at least one frame in the scene, resulting in a plurality of selected frames from the content, wherein the plurality of selected frames comprises at least one frame from each scene; generating, by the processor, an image wall based on the plurality of selected frames; activating, by the processor, a computer vision model based on the topic and the image wall; and determining, by the processor, a semantic meaning of the content based on the activation of the computer vision model.

11. The method of claim 10, wherein identifying the plurality of scenes comprises: identifying a plurality of scene changes, each scene change indicating a change from one scene to another scene along a timeline of the content.

12. The method of claim 10, wherein the visual data comprises video, each scene representing a portion of the video, and wherein identify ing at least one image in the scene comprises selecting at least one frame from the scene, the at least one frame corresponding to the at least one image.

13. The method of claim 12, further comprising: cropping and / or scaling the at least one image to reduce the size of the at least one image based on an image size limit of a computer vision model.

14. The method of claim 13, wherein the computer vision model has a maximum horizontal pixel dimension and a maximum vertical pixel dimension, and wherein cropping or scaling the at least one image comprises:(1) cropping and / or scaling the at least one image to reduce at least the horizontal dimension of the at least one image based on the maximum horizontal dimension, and / or (2) cropping and / or scaling the at least one image to reduce at least the vertical dimension of the at least one image based on the maximum vertical dimension.

15. The method of claim 10, further comprising: stitching each image in the image wall to conform to a resolution requirement of the image model.

16. The method of claim 10, further comprising: accessing a transcript of the content, the transcript being a transcription of spoken or written words of the content, wherein the one or more words or phrases are accessed from the transcript.

17. The method of claim 16, wherein identifying the topic comprise: performing autoregression analysis on the transcript.

18. The method of claim 10, wherein the computer vision model generates a listing of tags per image in the image wall, the method further comprising: determining semantic meaning of the visual data based on the listing of tags per image.

19. A non-transitory computer readable storage medium storing instructions that, when executed by a processor, program the processor to:access content comprising visual data; determine, based on a language model trained using autoregression to predict words or phrases in text, a topic of the content based on words or phrase in the content; identify a plurality of scenes from the visual data, each scene from among the plurality of scenes occupying a respective length of time in the visual data; select, for each scene from among the plurality of scenes, at least one frame in the scene, resulting in a plurality of selected frames from the content, wherein the plurality of selected frames comprises at least one frame from each scene; generate an image wall based on the plurality of selected frames; activate a computer vision model based on the topic and the image wall; and determine a semantic meaning of the content based on the activation of the computer vision model.

20. The non-transitory computer readable medium of claim 19, wherein to identify the plurality of scenes, the instructions when executed further programs the processor to: identify a plurality of scene changes, each scene change indicating a change from one scene to another scene along a timeline of the content.

Images